[ https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398593#comment-15398593 ]
Sidharta Seethana commented on YARN-5360: ----------------------------------------- To clarify a few things here - yes, customized images are required in some cases (especially in secure mode) to make apps work for certain users. This is a limitation we have to work with for the moment given the hadoop security model - it may not be reasonable or practical to drop it altogether except under controlled situations. Also, log aggregation does not work in secure mode if you drop "--user" (it works in non-secure mode, I think but I'll have to check the code/test again). Artifact deletion will not work if the artifacts are created as a different user in the docker container (artifact cleanup is done as the 'run as' user). In your first table above, the yarn/nobody case likely did not work because the sequenceiq image is based on centos (nobody uid=99) and the system you were testing on was not centos (ubuntu? nobody uid=65534). We have tested spark with other images on centos (I have test images on docker hub if you'd like to try). I am pretty sure [~templedf] has successfully run spark using the current implementation as well. ([~templedf] : please confirm) . If this the discussion here is now only about dropping the "--user" in certain cases, this is captured in YARN-4266. > Decouple host user and Docker container user > -------------------------------------------- > > Key: YARN-5360 > URL: https://issues.apache.org/jira/browse/YARN-5360 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn > Reporter: Zhankun Tang > Assignee: Zhankun Tang > > There is *a dependency between job submitting user and the user in the Docker > image* in LCE currently. For instance, in order to run the Docker container > as yarn user, we can choose set the > "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn > and leave > "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users" > default (true). Then LCE will choose yarn ( UID maybe 1001) as the user > running jobs. > LCE will mount the generated launch_container.sh (owned by the running job > user) and /etc/passwd (*current the code is mounting to container's > /etc/password, I think it's a mistake*) into the Docker container and > utilizes "docker run --user=<run_as_user>" option to get it done internally. > Mounting /etc/passwd to the container is a not good choice due to override > original users defined in Docker image. As far as I know, since Docker v1.8 > (or maybe earlier), the Docker run command "--user=" option accepts UID and > *when passing UID, the user does not have to exist in the container*. So we > could use UID instead of user name to construct the Docker run command to > eliminate the dependency that create the same user in the Docker image. This > enables LCE the ability to launch any Docker container safely regardless what > users in it. > But this is not enough to decouple host user and Docker container user. The > final solution we are searching for are focused on allowing users to run > their Docker images flexibly without involving dependencies of YARN and make > sure the container won't bring in security risk. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org