[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769904#comment-16769904 ] Zhankun Tang commented on YARN-8927: [~eyang] , Thanks for the review! [~ebadger] , I see. That makes sense to me. Thanks! > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Fix For: 3.3.0 > > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769483#comment-16769483 ] Hudson commented on YARN-8927: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15977 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15977/]) YARN-8927. Added support for top level Dockerhub images to trusted (eyang: rev 7c1b561e334f32cc0b5011fc52c47e0758fd47a9) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Fix For: 3.3.0 > > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16769435#comment-16769435 ] Eric Yang commented on YARN-8927: - [~tangzhankun] Tag can not contain '/'. I was referring to "docker tag" command to include '/' in repository name. Valid usage are: {code}docker tag centos:latest private/centos:latest{code} or {code}docker tag tensorflow/tensorflow:latest tensorflow:latest{code} If a admin run the second command. Tensorflow image becomes trusted if library keyword is given. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768880#comment-16768880 ] Zhankun Tang commented on YARN-8927: [~eyang] , sorry for the late reply. Just go through the above discussions. It makes sense to me that we check local image existence in Java layer since it controls pull or not. One thing I'm not sure if this is different from the Docker version. I cannot rename an image name with "/" in its tag in my Ubuntu VM. Docker version is "18.06.1-ce". [~ebadger] , can you do this in your environment? {code:java} root@master0-VirtualBox:/opt/code/hadoop# docker tag tangzhankun/repo1/sub1/tensorflow tensorflow:repo1/sub1 Error parsing reference: "tensorflow:repo1/sub1" is not a valid repository/tag: invalid reference format{code} > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768795#comment-16768795 ] Eric Yang commented on YARN-8927: - [~ebadger] Sure, I will remove the comment. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768778#comment-16768778 ] Eric Badger commented on YARN-8927: --- I'm ok with this going in as is given the addition of YARN-9306. {noformat} +// image name doens't contains "/" {noformat} [~eyang], could you change fix up this nit to {{doesn't contain}} before the commit? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768760#comment-16768760 ] Eric Yang commented on YARN-8927: - Patch 2 uses '/' to determine if the image is a top level image. It does not use '/' character to detect local image. If admin wants to authorize local image, he/she can tag local image with trusted registry prefix. As long as the trusted registry prefix does not have the same name as docker hub registry name, authorized local images are safe to use. If local image is named without '/' character, they are also allowed for now until YARN-9306 is addressed. It would take admin rights to tag local image without '/' character. The possibility of using library keyword to trigger unauthorized image to run is hard to accomplish. Patch 2 is good enough for me. +1 for patch 2. I will commit patch 2 if no objection. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16768482#comment-16768482 ] Eric Badger commented on YARN-8927: --- {quote} Chandni Singh is solving the docker image localization issues in YARN-9228. It may help to solve precheck of image existence in her story instead. {quote} I'm fine with moving this to another JIRA. I just don't want to preclude an environment where only local images are allowed. And doing the determination of whether an image is local or not based on the existence of a "/" character doesn't do that, since local images are perfectly allowed to contain the "/" character in their tag. I don't want to hold up this feature either, however. So maybe it's best to diverge into 2 different paths here. Keep this JIRA alive to deal only with the library keyword and have the library keyword only associated with dockerhub images. Then in another JIRA add a different keyword for local images. Because using the library keyword for local images in this state would not work out. I really don't like the idea of another keyword, since I hate making arbitrary special keywords, but I don't see another way around the issue here. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767790#comment-16767790 ] Eric Yang commented on YARN-8927: - [~ebadger] I think it's still admin mistake because the repository name can be preconfigured to a host in local domain which would have no chance to contact docker hub even if a repository is later setup to try to impersonate. YARN's trusted registry acl can avoid untrusted docker hub repository. The discussion is digressing. I agree that adding the local image white list can tighten security further for images without '/' characters or used. This jira can't solve docker run pulling remote image when image is absent or remote image name is identical to local image name. [~csingh] is solving the docker image localization issues in YARN-9228. It may help to solve precheck of image existence in her story instead. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767744#comment-16767744 ] Eric Badger commented on YARN-8927: --- This isn't an admin mistakenly naming their local image the same as a repository on dockerhub. The admin will name their local images something and then after that a nefarious actor will upload a malicious image to that same location in dockerhub. Unless you are assuming that dockerhub is to be a trusted source, which I don't think it can be. As for avoiding this issue by using a private repository, this is not possible as Docker refuses to remove docker.io from the default registry list (https://github.com/moby/moby/issues/33069). So docker.io will always be the fallback if the image does not exist locally. Again, I would love it if Docker would just allow for you to remove default registries or add a --no-pull flag or similar to the run command. But, since they are not and will not do those, we have to mitigate in other ways to avoid bad apples who can push malicious images to dockerhub. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767703#comment-16767703 ] Eric Yang commented on YARN-8927: - [~ebadger] I don't think there is a way to prevent docker run to pull a image that admin has mistakenly named local images that matches repository on docker hub, then having the image absent locally. The chance of this happening is rare and can be avoided by using private repository host/port to avoid contacting docker hub. I like to avoid conflating admin mistakes (usability problem) and actual security problem for this jira to move forward. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767645#comment-16767645 ] Eric Badger commented on YARN-8927: --- ARN-9184 deals with explicit pulls. However, docker will do an implicit pull during {{docker run}} if the image does not exist locally. YARN-9184 seems to deal with explicitly pulling (or not pulling) images before the container is launched. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767636#comment-16767636 ] Eric Yang commented on YARN-8927: - [~ebadger] {quote}If we are assuming that Dockerhub and any other default registry is untrusted (we should), then the assumption has to be that any image by any name can be published. Let's say I tag a local image as hadoop/myimage:latest on every node in my cluster. We have to assume that there could be a repo within the default registry named hadoop with an image named myimage:latest. This doesn't make my local image hadoop/myimage:latest any less of a local image, but it also means that there is an image in Dockerhub by the same name which will be pulled if, for whatever reason, my local image was deleted, not uploaded yet, etc.{quote} The last point is covered by YARN-9184. Can you confirm? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767583#comment-16767583 ] Eric Badger commented on YARN-8927: --- {quote} It seems if a user wants lcoal image "repoA/userA/imageA" to be allowed, he/she should configure "repoA/userA" in the "docker.trusted.registries"? I will try if this works and get back to you. {quote} It's not about wanting repoA/userA/imageA to be allowed. That is an easy problem to solve as you have described. The hard part is allowing repoA/userA/imageA to be allowed _only_ if it exists locally. {quote} And one thing worthing noting is that if YARN allows an image name, then Docker will check if it's local and prefer to run it before pulling from a hub. YARN's checking logic here seems duplicated work because if Docker can pull it and run. We can hardly say this "repoA/userA/imageA" is a real local image. {quote} If we are assuming that Dockerhub and any other default registry is untrusted (we should), then the assumption has to be that any image by any name can be published. Let's say I tag a local image as {{hadoop/myimage:latest}} on every node in my cluster. We have to assume that there could be a repo within the default registry named {{hadoop}} with an image named {{myimage:latest}}. This doesn't make my local image {{hadoop/myimage:latest}} any less of a local image, but it also means that there is an image in Dockerhub by the same name which will be pulled if, for whatever reason, my local image was deleted, not uploaded yet, etc. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767370#comment-16767370 ] Eric Yang commented on YARN-8927: - [~tangzhankun] when "library" is configured, and there is a local image named black. This is not a top level image. This image is trusted by default. In [~ebadger]'s environment, local trusted image is tagged with "repoA/imageA". Patch 002 breaks his trust list because top level images are trusted, but untagged image name black is also trusted. This is the reason that he ask for a local image white list to prevent local image like black to be trusted. Is this something that can be enhanced in the condition that checks for library and '/'? It would be possible to add a white list here to tighten security. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766696#comment-16766696 ] Zhankun Tang commented on YARN-8927: Just checked, if an image name is "repoA/userA/imageA", configure "repoA" and "repoA/userA" in "docker.trusted.registries" can both works. So it seems it doesn't need explicit logic to allow such named images? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766636#comment-16766636 ] Zhankun Tang commented on YARN-8927: [~eyang], [~ebadger] Thanks for the review! If a local image name contains "/" in it, it may not be considered as "top-level" image. It seems if a user wants lcoal image "userA/imageA" to be allowed, he/she should configure "user1" in the "docker.trusted.registries"? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16766493#comment-16766493 ] Eric Yang commented on YARN-8927: - {quote}If we see library/ in container-executor.cfg then we trust all local images.{quote} I am not sure how to identify if a image is local, if image contains '/' character. I think patch 002 will break [~ebadger]'s environment since local image names have '/' character in it. [~tangzhankun] any idea on how to fix this? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762825#comment-16762825 ] Eric Badger commented on YARN-8927: --- bq. Patch 002 implies that all local images are trusted as long as the image name does not have '/' character. This is not a good assumption. For example, in our internal clusters, we do local tagging of all images and all of our local tags have the "/" character in them. This is to signify the repository within the registry that they reside (albeit with the registry part chopped off). bq. I am unsure if another ACL is required to explicitly trust specific local images only I would prefer this behavior, because I don't think it is that much harder to implement. If we see {{library/centos:foobar}} in container-executor.cfg, then we strip off the {{library/}} part and trust local images related to the list using the suffix (e.g. {centos:foobar}}). If the image exists locally, then we continue with the launch. If it does not, then we fail out and do not pull If we see {{library/}} in container-executor.cfg then we trust all local images. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755512#comment-16755512 ] Eric Yang commented on YARN-8927: - [~tangzhankun] YARN-8955 is to skip docker pull, if the image already exist locally. Patch 002 implies that all local images are trusted as long as the image name does not have '/' character. On the surface, this is likely to be true. Admin privileges is required to run docker tag command to convert any image into a local image. I am unsure if another ACL is required to explicitly trust specific local images only. [~ebadger], you had some feedback before, and I like to revalidate if that requirement still necessary or the current implementation is good enough without hinders usability? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736696#comment-16736696 ] Hadoop QA commented on YARN-8927: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 1s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 34m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 58s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 45s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 44s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 25s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 71m 34s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | YARN-8927 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954087/YARN-8927-trunk.002.patch | | Optional Tests | dupname asflicense compile cc mvnsite javac unit | | uname | Linux b8bb012d9eb4 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 4894115 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/23017/testReport/ | | Max. process+thread count | 340 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/23017/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} >
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736650#comment-16736650 ] Zhankun Tang commented on YARN-8927: [~eyang] , Thanks for the review! Yeah, it doesn't consider the local image list in this JIRA. If I remember correctly, it will be handled in YARN-8955? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch, YARN-8927-trunk.002.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736478#comment-16736478 ] Eric Yang commented on YARN-8927: - [~tangzhankun] thank you for the patch. The handling of top level image seems to work properly. The test case uses container_executor_cfg instead of unused variable container_cfg. It looks like a good idea to use container_cfg, and free the local variable properly. Patch 001 doesn't include local image idea, are you planning to add trust local images? > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16736028#comment-16736028 ] Hadoop QA commented on YARN-8927: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 34m 45s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 38s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} cc {color} | {color:red} 0m 57s{color} | {color:red} hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 57s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 42s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 19m 53s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 71m 46s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f | | JIRA Issue | YARN-8927 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12954017/YARN-8927-trunk.001.patch | | Optional Tests | dupname asflicense compile cc mvnsite javac unit | | uname | Linux b3c27c838afb 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct 5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/patchprocess/precommit/personality/provided.sh | | git revision | trunk / d715233 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | cc | https://builds.apache.org/job/PreCommit-YARN-Build/23006/artifact/out/diff-compile-cc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/23006/testReport/ | | Max. process+thread count | 305 (vs. ulimit of 1) | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/23006/console | | Powered by | Apache Yetus 0.8.0 http://yetus.apache.org | This message was automatically generated. > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major
[jira] [Commented] (YARN-8927) Support trust top-level image like "centos" when "library" is configured in "docker.trusted.registries"
[ https://issues.apache.org/jira/browse/YARN-8927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16735941#comment-16735941 ] Zhankun Tang commented on YARN-8927: A draft patch WIP. Please comment in case the wrong direction. [~eyang] , [~ebadger] > Support trust top-level image like "centos" when "library" is configured in > "docker.trusted.registries" > --- > > Key: YARN-8927 > URL: https://issues.apache.org/jira/browse/YARN-8927 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Zhankun Tang >Assignee: Zhankun Tang >Priority: Major > Labels: Docker > Attachments: YARN-8927-trunk.001.patch > > > There are some missing cases that we need to catch when handling > "docker.trusted.registries". > The container-executor.cfg configuration is as follows: > {code:java} > docker.trusted.registries=tangzhankun,ubuntu,centos{code} > It works if run DistrubutedShell with "tangzhankun/tensorflow" > {code:java} > "yarn ... -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker -shell_env > YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=tangzhankun/tensorflow > {code} > But run a DistrubutedShell job with "centos", "centos[:tagName]", "ubuntu" > and "ubuntu[:tagName]" fails: > The error message is like: > {code:java} > "image: centos is not trusted" > {code} > We need better handling the above cases. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org