[
https://issues.apache.org/jira/browse/YARN-5360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15397027#comment-15397027
]
Zhankun Tang commented on YARN-5360:
------------------------------------
[~sidharta-s], thanks for pointing out this. Actually, different default
behaviors seems awkward for me too. But dropping --user in non-secure mode
seems no big issue breaking spark/MR per my testing. Could you please explain
this a little more?
I did some testings about the impacts dropping "--user" in non-secure mode
bring to current MR/Dockerized MR and spark (I don't find any official document
of Dockerized spark, please direct me to it if you know) job. Below are my
testing results:
||non-secure mode LCE With "--user"||
|Cluster Environment:|Hadoop branch-2.8|
|LCE Modifications:|*None*|
|Testing tool:| "Pi" application for MR and "SparkPi" for
spark|
|Testing result:
||yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user||yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users||submitting
user||run job user||"Pi" MR without Docker||Pi" MR with Docker image
"sequenceiq/hadoop-docker:latest"||"SparkPi" without Docker|||
| |yarn|(default)|yarn|yarn|(/)|(x)
"Unable to find user yarn, exit code 7"|(/)|
| |yarn|(default)|test|yarn|(/)|(x)
"Unable to find user yarn, exit code 7"|(/)|
| |yarn|false|test|test|(/)|(x) "Unable
to find user test, exit code 7"|(/)|
|
|(default)|(default)|yarn|nobody|(/)|(x) "launch_container.sh: Permission
denied. Unable to read from docker logs(ferror, feof): 0 1, exit code 126"|(/)|
||non-secure mode LCE Without "--user"||
|Cluster Environment:|Hadoop branch-2.8|
|LCE Modifications:|*Only remove the "--user" from mandatory options*
in constructor method of DockerRunCommand.java
(line 37)|
|Testing tool:| "Pi" application for MR and "SparkPi" for
spark|
|Testing result:
||yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user||yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users||submitting
user||expected run job user||"Pi" MR without Docker||Pi" MR with Docker image
"sequenceiq/hadoop-docker:latest"||"SparkPi" without Docker|||
| |yarn|(default)|yarn|yarn|(/)|(/)
(run as root)|(/)|
| |yarn|(default)|test|yarn|(/)|(/)
(run as root)|(/)|
| |yarn|false|test|test|(/)|(/) (run as
root)|(/)|
|
|(default)|(default)|yarn|nobody|(/)|(/) (run as root)|(/)|
> Decouple host user and Docker container user
> --------------------------------------------
>
> Key: YARN-5360
> URL: https://issues.apache.org/jira/browse/YARN-5360
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
> Reporter: Zhankun Tang
> Assignee: Zhankun Tang
>
> There is *a dependency between job submitting user and the user in the Docker
> image* in LCE currently. For instance, in order to run the Docker container
> as yarn user, we can choose set the
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user" to yarn
> and leave
> "yarn.nodemanager.linux-container-executor.nonsecure-mode.limit-users"
> default (true). Then LCE will choose yarn ( UID maybe 1001) as the user
> running jobs.
> LCE will mount the generated launch_container.sh (owned by the running job
> user) and /etc/passwd (*current the code is mounting to container's
> /etc/password, I think it's a mistake*) into the Docker container and
> utilizes "docker run --user=<run_as_user>" option to get it done internally.
> Mounting /etc/passwd to the container is a not good choice due to override
> original users defined in Docker image. As far as I know, since Docker v1.8
> (or maybe earlier), the Docker run command "--user=" option accepts UID and
> *when passing UID, the user does not have to exist in the container*. So we
> could use UID instead of user name to construct the Docker run command to
> eliminate the dependency that create the same user in the Docker image. This
> enables LCE the ability to launch any Docker container safely regardless what
> users in it.
> But this is not enough to decouple host user and Docker container user. The
> final solution we are searching for are focused on allowing users to run
> their Docker images flexibly without involving dependencies of YARN and make
> sure the container won't bring in security risk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
