[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16135750#comment-16135750 ] Eric Yang commented on YARN-4266: - [~ebadger] I think the idea is correct to use -u=$(id -u $(whoami)):$(id -g $(whoami)) to evaluate the running user inside container. Could you provide an updated patch that fixes the style check? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266.002.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16077356#comment-16077356
]
Hadoop QA commented on YARN-4266:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
| {color:green} 1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green} 1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 13m
14s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 9m
11s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s{color} | {color:green} trunk passed {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
7s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
49s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green} 1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} compile {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} cc {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javac {color} | {color:green} 5m
30s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 59s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 9 new 226 unchanged - 0 fixed = 235 total (was 226) {color} |
| {color:green} 1{color} | {color:green} mvnsite {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green} 1{color} | {color:green} findbugs {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green} 1{color} | {color:green} javadoc {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:green} 1{color} | {color:green} unit {color} | {color:green} 13m
41s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed.
{color} |
| {color:green} 1{color} | {color:green} asflicense {color} | {color:green} 0m
31s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 61m 30s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-4266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12875993/YARN-4266.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle cc |
| uname | Linux 29030cad3302 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26
18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7576a68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| findbugs |
https://builds.apache.org/job/PreCommit-YARN-Build/16317/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
|
| checkstyle |
https://builds.apache.org/job/PreCommit-YARN-Build/16317/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
|
| unit |
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076769#comment-16076769 ] Eric Badger commented on YARN-4266: --- bq. Commented over on that jira. It would be good if we could get some traction over there as it looks like that patch is pretty close to being done. Oops, must not have reloaded that tab in over a week. Didn't see [~luhuichun]'s response. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16076756#comment-16076756 ] Eric Badger commented on YARN-4266: --- [~shaneku...@gmail.com], sorry for the delay. I've been sidetracked a little bit. The patch that I have still has a few bugs, but I'd like to make sure that we agree on the approach before I move forward with cleaning it up and posting it here. bq. The intent to YARN-5534 is provide a mount white list, so I believe that should help here. The initial patch here could hard code the bind mount while we test and provide feedback. Hopefully we can leverage YARN-5534 before this is wrapped up. Commented over on that jira. It would be good if we could get some traction over there as it looks like that patch is pretty close to being done. bq. I don't think this is a requirement for the initial version. We could do a follow on effort to remove/reduce the need for the bind mounted socket for a known list of AMs, assuming the behavior can be changed in those AMs. This is true. I'm attempting to do my due diligence up front to see if there is an avenue to get MRAppMaster to work without mounting /var/run/nscd. I've been talking with [~daryn] offline who has done lots of work on UGI stuff and we're looking into solutions. One solution that he suggested was going to back to our original idea of doing the adduser/usermod hack during container startup. I don't like this as much as it only allows you to put the one user in the container and will fail any other user lookups. It also would never get user/group updates which might be relevant for long-running containers. And on top of that, it would be unnecessary in the face of bind-mounting /var/run/nscd. However, it does get over the initial obstacle of not being able to run without bind-mounting /var/run/nscd. Preferably, we can find a way to make the --user switch work with MRAppMaster, but if not maybe this is the way to go. Thoughts? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16066392#comment-16066392
]
Shane Kumpf commented on YARN-4266:
---
Thanks for taking the time to answer my questions, [~ebadger]. I'm very
interested in testing out the patch when it is ready.
{quote}Yea I'm really not a fan either. I would strongly prefer a better,
cleaner solution to this problem if there is one.{quote}
The intent to YARN-5534 is provide a mount white list, so I believe that should
help here. The initial patch could hard code the bind mount while we test and
provide feedback. Hopefully we can leverage YARN-5534 before this is wrapped up.
{quote}I'm looking into this. I'm hoping that we can get around this so that we
can optionally add the bind mount, but not require it for the --user option. I
have not yet tested other AMs.{quote}
I don't think this is a requirement for the initial version. We could do a a
follow on effort to remove/reduce the need for the bind mounted socket for a
known list of AMs, assuming the behavior can be changed in those AMs.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065414#comment-16065414
]
Eric Badger commented on YARN-4266:
---
bq. I'm not really a fan of hard coding bind mounts like we've done for
/sys/fs/cgroup if we can help it.
Yea I'm really not a fan either. I would strongly prefer a better, cleaner
solution to this problem if there is one.
bq. Are you aware of any security implications with this socket mounted
read-only in the container?
I haven't done any research into it, but I imagine bind-mounting a socket would
have more security implications than a regular directory. If we decide this
route is worth delving into, then I will do my due diligence of trying to
identify security risks and whether they are acceptable or not.
bq. Also, are there any clients that might be required to be installed in the
container depending on how nsswitch is configured?
If you bind mount /var/run/nscd then it will use the process listening on that
socket. This will end up using nsswitch on the host, not in the container. So
it would completely leverage host services, not services within the container.
This actually makes things nice for remote user lookups, because it can use the
host's cache. This means that the container won't have to do a remote user
lookup every time a container is started, assuming that it's in the host's user
cache, if it has one (such as nscd). So I don't believe that any extra services
would have to be installed within the container for this to be used, only on
the host. And if no service is listening on the nscd socket, then the user
lookup would do the user lookup like it normally would.
bq. Alternatively, why does MRAppMaster do the user lookup in this case? Is
there a way to remove that limitation? Are you finding other AM's have a
similar issue?
I'm looking into this. I'm hoping that we can get around this so that we can
optionally add the bind mount, but not require it for the {{--user}} option. I
have not yet tested other AMs.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065260#comment-16065260
]
Shane Kumpf commented on YARN-4266:
---
Thanks, [~ebadger]! It sounds like you are making good progress.
{quote} To do this, I propose mounting /var/run/nscd so that the docker
container can lookup users via the host according to whatever method is defined
in nsswitch.conf on the host.{quote}
I believe YARN-5534 will be required for this. I'm not really a fan of hard
coding bind mounts like we've done for /sys/fs/cgroup if we can help it. Are
you aware of any security implications with this socket mounted read-only in
the container? Also, are there any clients that might be required to be
installed in the container depending on how nsswitch is configured?
Alternatively, why does MRAppMaster do the user lookup in this case? Is there a
way to remove that limitation? Are you finding other AM's have a similar issue?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063931#comment-16063931
]
Eric Badger commented on YARN-4266:
---
I'm working on a new patch based on the one from [~tangzhankun] and
[~luhuichun], but would like some comments. In the patch I use the {{--user}}
option to set the uid:gid of the user in the {{docker run}} command. After
massaging the environment variables, installing the necessary packages in the
image, and etc. this almost works perfectly. However, MRAppMaster tries to do a
user lookup and fails because the user has no name, only a uid. So for jobs
that run through the MRAppMaster, they will fail when trying to use a specific
uid in the container. This doesn't necessarily apply to other jobs that could
be run in the docker container on YARN, so I think that this change is still an
improvement over the current implementation for some users.
But clearly, my goal here is to run through the MRAppMaster with an arbitrary
uid:gid pair. To do this, I propose mounting /var/run/nscd so that the docker
container can lookup users via the host according to whatever method is defined
in nsswitch.conf on the host. glibc will automatically go to the nscd socket to
see if there is a service listening. If a service is listening (such as nscd or
sssd on the host), then the lookup can leverage them to do the lookup. This
gives us the ability to do remote authentication via ldap. The downside, of
course, is that we're now bind mounting another directory, and a socket
nonetheless. So I'm very interested in comments on this approach.
cc [~shaneku...@gmail.com], [~templedf], [~vvasudev], [~sidharta-s]
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: luhuichun
> Attachments: YARN-4266.001.patch, YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987814#comment-15987814 ] Eric Badger commented on YARN-4266: --- [~luhuichun], looks like there's some compilation problems with the patch > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987771#comment-15987771
]
Hadoop QA commented on YARN-4266:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
0s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
in trunk has 5 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m
5s{color} | {color:red} hadoop-yarn in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 5s{color}
| {color:red} hadoop-yarn in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 51s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch
generated 4 new + 207 unchanged - 1 fixed = 211 total (was 208) {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
21s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
19s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 29s{color}
| {color:red} hadoop-yarn-api in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s{color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 8s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.yarn.conf.TestYarnConfigurationFields |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ac17dc |
| JIRA Issue | YARN-4266 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12864772/YARN-4266.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux e889b734bfdc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9460721 |
| Default Java | 1.8.0_121 |
|
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981311#comment-15981311 ] luhuichun commented on YARN-4266: - [~ebadger] Hi Eric, sorry for late response, we are working on another JIRA last week, here is the first patch > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: luhuichun > Attachments: YARN-4266.001.patch, YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15972986#comment-15972986 ] Eric Badger commented on YARN-4266: --- Hey [~tangzhankun], [~luhuichun], wondering if there's any update/what the status of this is. Do you have any sort of target date. Thanks! > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944212#comment-15944212 ] Zhankun Tang commented on YARN-4266: [~ebadger], Yes. We're currently working on it. [~luhuichun] will update the patch soon. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943925#comment-15943925 ] Eric Badger commented on YARN-4266: --- [~tangzhankun], have had had any time to look into this further? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926449#comment-15926449 ] Eric Badger commented on YARN-4266: --- [~tangzhankun], I'm ok waiting until next week. I just want to make sure that we keep the ball rolling and moving this forward instead of stagnating. I can test the patch out once you upload, hopefully sometime late next week. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15926362#comment-15926362 ] Zhankun Tang commented on YARN-4266: Sorry, [~ebadger], I'd like to help but I'm out of office until mid of the next week. Is this ok for you? If not, you can go ahead to do it. Thanks. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15924953#comment-15924953 ] Eric Badger commented on YARN-4266: --- [~tangzhankun], would you like to put up an updated version of your patch given the consensus that we seem to have around adding the off-by-default "--user=UID:GID" option (without the usermod stuff)? If not, I'd be happy to do the legwork. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901607#comment-15901607 ] Eric Badger commented on YARN-4266: --- Looks like I should have reloaded the page before commenting as I didn't see [~shaneku...@gmail.com]'s comment. It's unfortunate that docker's namespace remapping doesn't work for multiple users. I guess a single-user namespace emapping would help the problem, but not solve all use cases. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901562#comment-15901562 ] Eric Badger commented on YARN-4266: --- [~sidharta-s], I agree with your assessment. I don't see this "--user" workaround to be the longterm solution, especially if the goal is to allow users to supply their own arbitrary, untrusted images. As others have identified previously in this jira, I believe that the real solution is to use [user namespace remapping|https://success.docker.com/KBase/Introduction_to_User_Namespaces_in_Docker_Engine], which was introduced in Docker 1.10. However, that requires a more updated kernel (3.10) than I think most of us are on, especially in production. So, until then I think that allowing an arbitrary UID:GID (or even user:group) to enter the container will be sufficient (disabled by default, as you suggested). Though I believe that containers working in this way are under the big assumption that the image is trusted and well-crafted, which is necessary until we figure out the user remapping issue, resolve security concerns, etc. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15901534#comment-15901534
]
Shane Kumpf commented on YARN-4266:
---
I took another look at the progress being made on user namespaces in Docker and
as far as I can tell, the story remains the same. I echo [~sidharta-s], it just
doesn't appear there is a solution here that will work for all container types.
As [~templedf] pointed out, "Modifying the container is not a valid
alternative to modifying the container", but we are limited on options here. :)
As it appears the proposed solution will solve the problem for a class of
container types, I'm +1 on adding the UID/usermod approach as an optional
solution. Note that this solution won't help for official docker hub images
such as postgres and apache without some sort of setuid wrapper, so we'll need
to continue to discuss how we handle those.
I do believe that {{docker logs}} is worth exploring as a means of reducing or
eliminating the writable bind mounted directories. We could explore read-only
mounts for the various caches. It seems the biggest hurdle there will be the
secure tokens, but read-only may work here as well. Anyone already thought
about this far enough to have a story for the tokens? Should we open a new
ticket to discuss this approach?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15900256#comment-15900256
]
Sidharta Seethana commented on YARN-4266:
-
Based on the discussions in this JIRA and on YARN-5360, it looks like all we
have are less than ideal choices. Like I mentioned on YARN-5360, using the uid
has readability issues and it still wouldn’t guarantee that an image would work
correctly. In my opinion, we shouldn’t be adding *more* requirements on images
- the whole objective of this jira was to remove a requirement where possible
({{--user}}). launch_container.sh already uses bash, ln, cp, chmod, ls, find.
To this list we are considering adding usermod, su, getent and so on. In
addition to this we are considering making (expensive) changes to a container
prior to launching the application process - usermod only changes the files in
a user’s home directory and even then we still have no way of predicting how
long this operation would take - making application (process) launch time
unpredictable. IMO, This is not the direction we should be going in.
In the interest of making some progress, perhaps we could add support for
optionally using {{--user=:}}(turned off by default). A subset of
images that wouldn’t otherwise work, would be usable because of this change -
for example : images that don’t have the user being specified (say foo) but
would otherwise work with an arbitrary user (i.e the values supplied in
{{--user=:}} don’t matter).
I might have said this on other JIRAs and I’ll repeat here : docker containers
and applications using them are just one category of workloads that are going
to be run on a production YARN cluster. While we would like to use as much of
the power and flexibility that docker provides, we have to do this with due
consideration given to existing YARN/hadoop paradigms - security model
(users/groups/permissions), localization, log aggregation and so on.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15899828#comment-15899828
]
Eric Badger commented on YARN-4266:
---
[~tangzhankun], thanks for pointing that out. I hadn't seen that conversation.
It seems that the major issue with using --user=UID:GID is that there is no
username. But is there any reason that we can't just add in an environment
variable to the docker run command that is set to the username and then run a
usermod to change the username of the associated UID? Usernames are just
cosmetic and everything is done via UIDs, so I don't think it makes sense to
run the docker container based on a username.
Something like:
{{docker run --user=2000 -e USERNAME=\*username crafted in code\*}}
And then in the container startup command (with the container running as root):
{{usermod -l $USERNAME $(getent passwd "1001" | cut -d: -f1) && su $USERNAME}}
There are probably more efficient ways to do this, but this is just a general
idea and proof of concept.
The main problem that I can see with this method is if there is already a user
in the image associated with the UID of the user on the host. In that case, we
would need to remap the UID of the user in the image to something different
before we could do the usermod (or else we would have potential permissions
issues inside the container). However, this would also be easy to do.
[~sidharta-s], [~templedf], [~vvasudev], [~zyluo], you were all very active on
YARN-5360. Do you have any thoughts on the approach above given my explanation?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15896081#comment-15896081 ] Zhankun Tang commented on YARN-4266: Great thanks to [~ebadger] and [~templedf]. Sorry for the late reply. [~ebadger], I had proposed the "--user=UID:GID" way at the very beginning. Please refer to [here|https://issues.apache.org/jira/browse/YARN-5360?focusedCommentId=15374621=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15374621] for the details. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15895132#comment-15895132
]
Eric Badger commented on YARN-4266:
---
[~templedf], I think you're misunderstanding what I was saying. When we
instantiate the {{DockerRunCommand}}, we give it a user to run as. In this user
remapping case, we've been setting that user to root so that we have
permissions to perform the usermod remapping in the container. I'm saying,
however, that we could set that use to the UID and GID of the user that
submitted the job (i.e. {{runAsUser}} in the code).
So let's say user foo:1000:1000 submitted a job. The NM will create the
launch_container.sh script assuming user foo. Then we will go to launch the
docker container. When we instantiate the {{DockerRunCommand}}, we would pass
it the output of {{id -u}} and {{id -g}}. This sort of thing is already being
done in [~tangzhankun]'s patch to get {{targetUID}}. The result would give us a
command that looks like: {{docker run --user=1000:1000 ...}}. There isn't a
security hole here that I can see because the user in the container will have
the same UID/GID as the user that submitted the job. Inside the container, the
username associated with the UID doesn't really matter. Outside of the
container, everything written by the user in the container will have the same
UID. A downside would be that the username inside of the container isn't
meaningful and could be potentially very misleading to those who are unaware of
how this is all being done.
I've tested the --user=UID:GID option locally on a single-node cluster and have
been successful. Files/logs/etc. written in the container using are owned by
the user that submitted the job, which is the UID:GID given in the --user
option (foo:1000:1000 in the example above).
There also isn't a problem with usernames being numbers (which the image could
map to arbitrary UID/GIDs) because docker interprets all numbers in the --user
option as UID/GIDs. I tested this locally to make sure. So even if there is a
user named "2000" (with UID != 2000), the command {{docker run --user=2000}}
will create a new user with UID 2000.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894809#comment-15894809 ] Daniel Templeton commented on YARN-4266: Thanks for testing the patch, [~ebadger]! The reason we can't just specify the user in the run command is that the NM will write the launch script and all the other data for the job into the working directory owned by the job owner. We then mount that directory into the Docker container and exec it. If we set the user in the run command, the Docker container wouldn't have the permissions to access the directory or launch the script. If we tried to write the working directory et al as the user ID we intend to use in the Docker container, we open a potential security hole, because the user with that ID on the NM would be able to access it. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf, > YARN-4266-branch-2.8.001.patch > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894748#comment-15894748
]
Eric Badger commented on YARN-4266:
---
Also, as an aside, is there any reason that we want to go with the usermod
approach instead of leveraging docker to do this for us in the run command?
https://docs.docker.com/engine/reference/run/#user
We can still use the {{--user}} flag, but instead of passing a username we can
just pass the UID/GID of runAsUser. According to the docker documentation, it
will create the user if it doesn't exist, which means that we don't need to
have a predefined user, plus we don't need to start the container as root.
Until we get to user namespace remapping as [~shaneku...@gmail.com] eluded to
earlier, it seems to me that this would be a less hacky way to get around the
permissions issues. Thoughts?
cc: [~tangzhankun], [~sidharta-s], [~dan...@cloudera.com]
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894675#comment-15894675
]
Eric Badger commented on YARN-4266:
---
[~tangzhankun], I tried out the most recent patch that you put up and I have a
few comments.
{noformat}
+ if (disableUserRemapping) {
{noformat}
{noformat}
+ if (disableUserRemapping && targetUID != null) {
{noformat}
- Shouldn't {{disableUserRemapping}} be negated in these statements? We want to
do the remapping if {{disableUserRemapping}} is *not* disabled.
{noformat}
+containerPredefinedUser = getDockerImageInfo("{{.Config.User}}",
imageName, containerIdStr);
{noformat}
- There is no check to see whether {{containerPredefinedUser}} actually got set
to anything. It's possible for docker inspect to not return a predefined user.
In this case, we will be unable to do remapping and the usermod command will
fail because the user will be blank.
{noformat}
+//get runAsUser's UID for container to usermod when init
+if (!containerPredefinedUser.equals("root")) {
+ targetUID = getLocalUid(runAsUser);
+}
{noformat}
- I think checking {{containerPredefinedUser}} misses some cases here. You may
still want the container to be run as a different user even if the predefined
user is root. If we don't remap when the predefined user is root, then anything
written out to shared data volumes will have messed up permissions outside of
the container.
{noformat}
+String cmd = "\"usermod -o -u " + targetUID + " " +
containerPredefinedUser
++ " && su " + containerPredefinedUser + " bash -c '"
{noformat}
- It's not guaranteed that the predefined user has /bin/bash shell permissions.
So it may be prudent to add a {{-s /bin/bash}} to the usermod command.
Making the above changes I've been able to successfully submit and run jobs as
multiple different users without permissions issues. The only necessity seems
to be that there be a predefined user in the image that is being used.
Additionally, this usermod approach doesn't currently deal with group
permissions at all, which could be an issue especially in multi-tenant
clusters.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf,
> YARN-4266-branch-2.8.001.patch
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15736448#comment-15736448 ] Daniel Templeton commented on YARN-4266: I don't have an issue taking the usermod path, assuming that we're only changing the UID on the home dir and that we clearly document what we do and what to do if you want to opt out. That we take the usermod path now does not preclude continued work to improve the process, such as trying to make the Docker logs approach work. It would be bad if we end up with more than one way to solve the problem that we expose to the user, but I see no issue with evolving the implementation if we find a better solution. The end users shouldn't care if a later implementation reduces the constraints or improves the process. That said, I think it behooves us to at least investigate whether we can make the docker logs approach work. I know we've done some initial research, but have we gone far enough to know for sure how viable it is? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266-branch-2.8.001.patch, YARN-4266.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687059#comment-15687059 ] Shane Kumpf commented on YARN-4266: --- Thanks for the design document and discussion on the approaches, [~tangzhankun]! I think we can agree that there are challenges with all of the proposed approaches, but it seems an ideal approach may not exist today. As you and others have called out; 3.1 has security implications, 3.2 could introduce significant overhead, 3.3 depends on docker logs which is error prone and handling of tokens is an unknown. Given these approaches, on the surface, 3.3 seems like the least invasive wrt container changes. Eliminating the writable bind mounts may also make security easier to grok. Getting the token into the container doesn't seem all that difficult to address. How do others feel about 3.3? Not to hijack, but on a related note, I still believe user namespace remapping will be our future solution here. User namespace remapping would allow us to map the root user in the container, to the run as user on the host, eliminating many of the issues. I revisited this feature this morning in hopes it had evolved in the last couple of releases of Docker, but unfortunately it hasn't. The current user namespace remapping feature in docker can only be applied to a single user and is set at the daemon level, which will not work for us in both non-secure and secure modes. I believe it would currently be possible to support user namespace remapping for non-secure mode, but not both. Many issues are opened on the docker github requesting per container user namespace remapping, but the sharing of image layers makes this non-trivial to add. I really don't like the idea of varying approaches for secure and non-secure mode, but I would be happy to work on this approach for non-secure containers if others feel it is worth pursuing. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: YARN-4266-branch-2.8.001.patch, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15666332#comment-15666332
]
Zhankun Tang commented on YARN-4266:
[~sidharta-s], Great thanks for reviewing this!
{quote}
Like I mentioned in a earlier comment, the usermod operation only makes changes
to the home directory and not elsewhere in the image. Modifying the rest of the
image is not scalable and could significantly slow down the launch of every
container where we go down this path.
{quote}
Yes. Agree with this. This is a drawback that we cannot avoid at present.
{quote}
Running this usermod operation also requires that launch_container.sh be
launched as a privileged user. Also, note that running docker run --pid=host …
bash ../launch_container.sh $newUID $containerUsername does not guarantee that
launch_container.sh as described here will work correctly - if the image has a
‘USER’ directive, launch_container.sh will be run as that user and that user
might not have privileges to run a usermod operation.
{quote}
You might missed the part in the patch that we'll use "\-\-user=root" to
guarantee successful "usermod". We first inspect the Docker image, if it setup
a non-root user and wants to run with it, we'll use "--user=root". If the setup
user in image is root, we'll just let it go.
{quote}
In addition, I don’t believe we should be using —pid=host. This exposes other
containers’s processes into this container - which will break isolation and
possibly behavior for certain applications (applications that assume a single
instance is running on a ’node’, for example).
{quote}
thanks for pointing this. I forget to delete this when I'm trying different
implementation(sudo issue if I remember correctly). I have a double-check and
--pid=host is not needed.
{quote}
Lastly, adding more commands that run inside the container (usermod in this
case) adds even more requirements for the docker image being launched : we
already require that bash, find, ls etc be present in the image.
{quote}
This usermod is installed by default in most distributions I guess. Since we
already require several commands in the image, why cannot we document this too?
IMO, this is the light-weight way that can work securely and won't break down
the log staff. I got some complaint about current user remapping from customer.
Anyway, this JIRA is an important feature to make YARN a good supporter for
container(Docker and others maybe) and possibly not only big data Docker
images. We should involve more people on this. [~templedf], [~vvasudev],
[~shaneku...@gmail.com], [~zyluo]?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15665610#comment-15665610
]
Sidharta Seethana commented on YARN-4266:
-
IMO, the usermod approach described in the design document is error prone and
likely to break in various ways.
* Like I mentioned in a earlier comment, the usermod operation only makes
changes to the home directory and not elsewhere in the image. Modifying the
rest of the image is not scalable and could significantly slow down the launch
of every container where we go down this path.
* Running this usermod operation also requires that launch_container.sh be
launched as a privileged user. Also, note that running {{docker run --pid=host
… bash ../launch_container.sh $newUID $containerUsername}} does not guarantee
that launch_container.sh as described here will work correctly - if the image
has a ‘USER’ directive, launch_container.sh will be run as that user and that
user might not have privileges to run a usermod operation.
* In addition, I don’t believe we should be using —pid=host. This exposes
other containers’s processes into this container - which will break isolation
and possibly behavior for certain applications (applications that assume a
single instance is running on a ’node’, for example).
* Lastly, adding more commands that run inside the container (usermod in this
case) adds even more requirements for the docker image being launched : we
already require that bash, find, ls etc be present in the image.
I can’t think of a way where we can find a generic solution for disabling —user
that will work for all (or even a large number of) scenarios while still being
in line with YARN’s security model/log aggregation etc., . (I’d be happy to be
proven wrong here). I think we need to acknowledge this and find a set of
canonical use cases that we want to support - and see how we can enable each
one of them. Couple of examples off the top of my head : 1) Spark/MRv2 - IMO,
it maybe easier to support these because they are already first class YARN
applications. 2) Apache/httpd based images - more digging to do there.
Thoughts ? I apologize for my spotty availability on this JIRA - I’ll try to
provide more prompt responses/feedback in the future.
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581196#comment-15581196
]
Zhankun Tang commented on YARN-4266:
[~sidharta-s] Although we can alleviate it by "find / -user -exec
chown -h {} \;", I'm afraid this will cost overhead if nothing outside
the user's home directory needs ownership changes. Or we can just remind end
user about this limitation if we don't want this overhead
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[
https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15576903#comment-15576903
]
Sidharta Seethana commented on YARN-4266:
-
Usermod seems to be of limited use. From usermod's man page :
{code}
-u, --uid UID
The new numerical value of the user's ID.
This value must be unique, unless the -o option is used. The value
must be non-negative.
The user's mailbox, and any files which the user owns and which are
located in the user's home directory will have the file user ID changed
automatically.
The ownership of files outside of the user's home directory must be
fixed manually.
No checks will be performed with regard to the UID_MIN, UID_MAX,
SYS_UID_MIN, or SYS_UID_MAX from /etc/login.defs.
{code}
If nothing outside the user's home directory is updated, this is likely to
break many images that use custom/non-root users ?
> Allow whitelisted users to disable user re-mapping/squashing when launching
> docker containers
> -
>
> Key: YARN-4266
> URL: https://issues.apache.org/jira/browse/YARN-4266
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Reporter: Sidharta Seethana
>Assignee: Zhankun Tang
> Attachments: YARN-4266-branch-2.8.001.patch,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf,
> YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v3.pdf
>
>
> Docker provides a mechanism (the --user switch) that enables us to specify
> the user the container processes should run as. We use this mechanism today
> when launching docker containers . In non-secure mode, we run the docker
> container based on
> `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in
> secure mode, as the submitting user. However, this mechanism breaks down with
> a large number of 'pre-created' images which don't necessarily have the users
> available within the image. Examples of such images include shared images
> that need to be used by multiple users. We need a way in which we can allow a
> pre-defined set of users to run containers based on existing images, without
> using the --user switch. There are some implications of disabling this user
> squashing that we'll need to work through : log aggregation, artifact
> deletion etc.,
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15513926#comment-15513926 ] Daniel Templeton commented on YARN-4266: After some discussion and thought, I think that 3.2 is OK as long as the user has the option to turn it off. In other words, the contract is that either the user takes care of making sure the container has the required user, or let YARN do it for you. It might be worth exploring how far dockers logs could take us in dealing with the logging end of the problem. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf, > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping_v2.pdf > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498501#comment-15498501 ] Zhankun Tang commented on YARN-4266: Thanks for the review and poiting out the GID thing. [~templedf]. I have done testings based on quick implementations for the two ways of option 3.2. Based on testings, a generic launch_container.sh serving all scenarios seems not possible. So leaving launch_container.sh and modify it only in LCE when white-listed user is preferred. This ensure that current use cases won't be affected. The only known limitation of this solution is that we need the Docker image to package "sudo" command in it because we leverage it to switch user in container at run-time. But as you mentioned, there are still more concerns for my preferred solution: * *What if the new UID/GID is already in use in the container?* The usermod and groupmod commands both has an "-o" option that allow using duplicate (non-unique) UID/GID. So we are able to change the UID and GUID to what we want without failure. * *How can we run a container without modification?* The answer depends on the container application type(One thing to note is that if we allow a container run as root for white-listed users, we don't need to modify container. This needs discussion too but I guess it's hard to get approved). ** For applications in container that have dependencies on local host, like MRv2 application: The applications would almost certain encounter the log file write permission and the secure token read permission issues due to the "bind-mounted" directory. ** For applications in container don't have dependencies on local host, like web application: We seem need only to solve the log write permission during my testing (not sure for the log aggregation). Or find another way which don't persist log in local host. * To sum up, if we don't want to change container at all, *the known blocking issues here are how to collect log from container and how to feed the container secure token in a different way from "bind-mount" host directory*. IMO, it's nearly impossible because the only way to connect container and host seems "bind-mount" or it will involve modification to container. But again, current MRv2/spark in container already needs UID modification in Docker images, requirements of "sudo" installed in Docker images is not unacceptable to enable this white-listed user as a light weight solution. Anyway, I'm open to any better solution. Ideas? [~templedf], [~vvasudev], [~sidharta-s] > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15497423#comment-15497423 ] Daniel Templeton commented on YARN-4266: I don't really like either option. 3.1 is a no-go for security reasons. 3.2 seems risky. First, just running usermod isn't enough, as explained in https://muffinresearch.co.uk/linux-changing-uids-and-gids-for-user/ Second, how can you guarantee that the new UID isn't already in use in the container? Third, the reason we're having this discussion is because it's desirable to run a container without modification. Modifying the container is not a valid alternative to modifying the container. I think we still have some brainstorming to do. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > Attachments: > YARN-4266_Allow_whitelisted_users_to_disable_user_re-mapping.pdf > > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425829#comment-15425829 ] Zhankun Tang commented on YARN-4266: Not yet. I'll submit the proposal asap. Thanks. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Zhankun Tang > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422403#comment-15422403 ] Sidharta Seethana commented on YARN-4266: - [~tangzhankun], please feel free to take this over. Do you have a proposal for dealing with logs, directories etc? > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4266) Allow whitelisted users to disable user re-mapping/squashing when launching docker containers
[ https://issues.apache.org/jira/browse/YARN-4266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15422307#comment-15422307 ] Zhankun Tang commented on YARN-4266: [~sidharta-s], Any progress/update on this? We would like to help pushing this if you will. > Allow whitelisted users to disable user re-mapping/squashing when launching > docker containers > - > > Key: YARN-4266 > URL: https://issues.apache.org/jira/browse/YARN-4266 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana > > Docker provides a mechanism (the --user switch) that enables us to specify > the user the container processes should run as. We use this mechanism today > when launching docker containers . In non-secure mode, we run the docker > container based on > `yarn.nodemanager.linux-container-executor.nonsecure-mode.local-user` and in > secure mode, as the submitting user. However, this mechanism breaks down with > a large number of 'pre-created' images which don't necessarily have the users > available within the image. Examples of such images include shared images > that need to be used by multiple users. We need a way in which we can allow a > pre-defined set of users to run containers based on existing images, without > using the --user switch. There are some implications of disabling this user > squashing that we'll need to work through : log aggregation, artifact > deletion etc., -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
