[
https://issues.apache.org/jira/browse/MAPREDUCE-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18093258#comment-18093258
]
Jan Van Besien commented on MAPREDUCE-7540:
-------------------------------------------
I meanwhile realized that this fix is necessary (I think) but not sufficient.
We'll also need a fix for HADOOP-19907.
> JDK24 subject propagation case missed in CommitterEventHandler
> --------------------------------------------------------------
>
> Key: MAPREDUCE-7540
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7540
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 3.5.0
> Reporter: Jan Van Besien
> Priority: Major
>
> HADOOP-19670 converted {{CommitterEventHandler}} to use
> {{{}SubjectInheritingThread{}}}, but the backing {{ThreadFactory}} that
> creates them is only installed when the job classloader is enabled:
> {code:java}
> // CommitterEventHandler.serviceStart()
> ThreadFactoryBuilder tfBuilder = new ThreadFactoryBuilder()
> .setNameFormat("CommitterEvent Processor #%d");
> if (jobClassLoader != null) {
> // if the job classloader is enabled, we need to use the job classloader
> // as the thread context classloader (TCCL) of these threads in case the
> // committer needs to load another class via TCCL
> ThreadFactory backingTf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> Thread thread = new SubjectInheritingThread(r);
> thread.setContextClassLoader(jobClassLoader);
> return thread;
> }
> };
> tfBuilder.setThreadFactory(backingTf);
> }
> ThreadFactory tf = tfBuilder.build();
> {code}
> When {{mapreduce.job.classloader}} is not enabled,
> {{ThreadFactoryBuilder.build()}} falls back to
> {{{}Executors.defaultThreadFactory(){}}}, so the "CommitterEvent Processor"
> pool threads are plain threads.
> On JDK 24+ (SecurityManager permanently disabled, JEP 486) plain threads no
> longer inherit the Subject bound by {{{}Subject.callAs{}}}. So inside
> {{{}CommitterEventHandler$EventProcessor{}}},
> {{UserGroupInformation.getCurrentUser()}} finds no Subject and falls back to
> the login user. As a result {{OutputCommitter.setupJob()}} / {{commitJob()}}
> perform their HDFS operations as the NodeManager's OS user (e.g.
> {{{}yarn{}}}) instead of the job submitter.
> Observed on Hadoop 3.5.0 running on JDK 25 (see stack trace below).
> Note the asymmetry in the stack trace: the NameNode side shows the patched
> {{SubjectInheritingThread}} frames from HADOOP-19668, while the AM side
> bottoms out in a bare {{Thread.run}} with no Subject wrapper frames.
> {noformat}
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=yarn, access=WRITE,
> inode="/tmp/lily/crunch/crunch-1246119727/p27":airflow:lily:drwxrwxr-x
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:663)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:503)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermissionWithContext(FSPermissionChecker.java:527)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:397)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1974)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1955)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1914)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:62)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3568)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1176)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideT
> ranslatorPB.java:742)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNa
> menodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:631)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:599)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine2$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine2.java:583)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1228)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1312)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:1227)
> at
> java.base/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
> at
> java.base/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
> at java.base/java.lang.ScopedValue$Carrier.call(ScopedValue.java:419)
> at java.base/javax.security.auth.Subject.callAs(Subject.java:331)
> at
> org.apache.hadoop.security.authentication.util.SubjectUtil.callAs(SubjectUtil.java:242)
> at
> org.apache.hadoop.security.authentication.util.SubjectUtil.doAs(SubjectUtil.java:314)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1958)
> at org.apache.hadoop.ipc.Server$Handler.work(Server.java:3279)
> at
> org.apache.hadoop.util.concurrent.SubjectInheritingThread$1.run(SubjectInheritingThread.java:203)
> at
> org.apache.hadoop.util.concurrent.SubjectInheritingThread$1.run(SubjectInheritingThread.java:199)
> at
> java.base/jdk.internal.vm.ScopedValueContainer.callWithoutScope(ScopedValueContainer.java:162)
> at
> java.base/jdk.internal.vm.ScopedValueContainer.call(ScopedValueContainer.java:147)
> at java.base/java.lang.ScopedValue$Carrier.call(ScopedValue.java:419)
> at java.base/javax.security.auth.Subject.callAs(Subject.java:331)
> at
> org.apache.hadoop.security.authentication.util.SubjectUtil.callAs(SubjectUtil.java:242)
> at
> org.apache.hadoop.security.authentication.util.SubjectUtil.doAs(SubjectUtil.java:275)
> at
> org.apache.hadoop.util.concurrent.SubjectInheritingThread.run(SubjectInheritingThread.java:199)
> at
> java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:62)
> at
> java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
> at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:483)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:121)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:88)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2557)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2531)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1507)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$27.doCall(DistributedFileSystem.java:1504)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirsInternal(DistributedFileSystem.java:1521)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.mkdirs(DistributedFileSystem.java:1496)
> at org.apache.hadoop.fs.FileSystem.mkdirs(FileSystem.java:2497)
> at
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.setupJob(FileOutputCommitter.java:361)
> at
> org.apache.crunch.io.CrunchOutputs$CompositeOutputCommitter.setupJob(CrunchOutputs.java:300)
> at
>
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.handleJobSetup(CommitterEventHandler.java:256)
> at
> org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler$EventProcessor.run(CommitterEventHandler.java:236)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1090)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:614)
> at java.base/java.lang.Thread.run(Thread.java:1474)
> {noformat}
> {*}fix{*}: always install the backing factory; only the classloader
> assignment should be conditional:
> {code:java}
> ThreadFactory backingTf = new ThreadFactory() {
> @Override
> public Thread newThread(Runnable r) {
> Thread thread = new SubjectInheritingThread(r);
> if (jobClassLoader != null) {
> thread.setContextClassLoader(jobClassLoader);
> }
> return thread;
> }
> };
> tfBuilder.setThreadFactory(backingTf);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
