wuchang created YARN-6543:
-----------------------------
Summary: yarn application's privilege is determined by yarn
process creator instead of yarn application user.
Key: YARN-6543
URL: https://issues.apache.org/jira/browse/YARN-6543
Project: Hadoop YARN
Issue Type: Bug
Reporter: wuchang
My application is a pyspark application which is impersonated by user 'wuchang'
My application infomation is :
{code}
Application Report :
Application-Id : application_1493004858240_0007
Application-Name : livy-session-6
Application-Type : SPARK
User : wuchang
Queue : root.wuchang
Start-Time : 1493708942748
Finish-Time : 0
Progress : 10%
State : RUNNING
Final-State : UNDEFINED
Tracking-URL : http://10.120.241.82:34462
RPC Port : 0
AM Host : 10.120.241.82
Aggregate Resource Allocation : 4369480 MB-seconds, 2131 vcore-seconds
Diagnostics :
{code}
And the process is :
{code}
appuser 25454 25872 0 15:09 ? 00:00:00 bash
/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/default_container_executor.sh
appuser 25456 25454 0 15:09 ? 00:00:00 /bin/bash -c /home/jdk/bin/java
-server -Xmx1024m
-Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
'-Dspark.ui.port=0' '-Dspark.driver.port=40969'
-Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError='kill %p'
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://[email protected]:40969 --executor-id 2 --hostname
10.120.241.18 --cores 1 --app-id application_1493004858240_0007
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
1>
/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stdout
2>
/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004/stderr
appuser 25468 25456 2 15:09 ? 00:00:09 /home/jdk/bin/java -server
-Xmx1024m
-Djava.io.tmpdir=/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/tmp
-Dspark.ui.port=0 -Dspark.driver.port=40969
-Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493004858240_0007/container_1493004858240_0007_01_000004
-XX:OnOutOfMemoryError=kill %p
org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
spark://[email protected]:40969 --executor-id 2 --hostname
10.120.241.18 --cores 1 --app-id application_1493004858240_0007
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/__app__.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-api-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-rsc-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/netty-all-4.0.29.Final.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/commons-codec-1.9.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-core_2.11-0.3.0-SNAPSHOT.jar
--user-class-path
file:/data/data/hadoop/tmp/nm-local-dir/usercache/wuchang/appcache/application_1493004858240_0007/container_1493004858240_0007_01_000004/livy-repl_2.11-0.3.0-SNAPSHOT.jar
appuser 26936 25846 0 15:16 pts/0 00:00:00 grep --color=auto
application_1493004858240_0007
{code}
The main problem is that the application user is "wuchang" , but the yarn
application is created by my OS super-user "appuser" , so , the privilege
becomes the problem. My code always run as the privilege of appuser instead of
"wuchang".
For example , below is the pyspark code:
{code}
import os
os.system("hadoop fs -rm -r /user/appuser/test.dat")
{code}
user "wuchang" should not have privilege to remove the file test.dat which
located in the home directory of appuser. But since the yarn application
process is created by "appuser", it does, although the yarn application user is
"wuchang".
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
