[
https://issues.apache.org/jira/browse/TEZ-2364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Seth updated TEZ-2364:
--------------------------------
Attachment: TEZ-2364.1.txt
This happens because the staging dir contains the parameter user.name - which
gets resolved based on the System property user.name.
On the client, it resolves to the user running the job. On the AM, since this
is a non secure cluster, and without LCE, this resolves to the user running the
AM process.
Patch resolves these config properties on the client, before sending it over to
the AM.
Another option here would be to stop using ${user.name}, and instead explicitly
add the user via the UGI when constructing the staging directory.
[~hitesh] - please review.
> 'joindatagen' AM picks wrong staging dir
> ----------------------------------------
>
> Key: TEZ-2364
> URL: https://issues.apache.org/jira/browse/TEZ-2364
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Yesha Vora
> Assignee: Siddharth Seth
> Fix For: 0.7.0
>
> Attachments: TEZ-2364.1.txt
>
>
> joindatagen application is run as hrt_qa user. and tez.staging-dir is set as
> {noformat}'/tmp/${user.name}/staging'{noformat}
> Here, AM tries to access "/tmp/yarn/staging" dir instead
> "/tmp/hrt_qa/staging".
> {code}
> 2015-04-23 21:09:13,032 INFO [main] app.DAGAppMaster: Comparing client
> version with AM version, clientVersion=0.7.0.2.3.0.0-1716,
> AMVersion=0.7.0.2.3.0.0-1716
> 2015-04-23 21:09:14,653 INFO [main] service.AbstractService: Service
> org.apache.tez.dag.app.DAGAppMaster failed in state INITED; cause:
> org.apache.tez.dag.api.TezUncheckedException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=hrt_qa, access=EXECUTE, inode="/tmp/yarn/staging":yarn:hdfs:drwx------
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
> at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1698)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3849)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1005)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:843)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:969)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
