[
https://issues.apache.org/jira/browse/YARN-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013167#comment-17013167
]
Eric Badger commented on YARN-9292:
-----------------------------------
{quote}
Doesn't the container know what image it was started with in its environment?
ServiceScheduler runs as part of application master for YARN service. YARN AM
is not containerized. Docker command to resolve image digest id happens prior
to any docker container is launched. The lookup for the docker image is from
the node which AM is running. We use the sha256 digest from AM node as
authoritative signature to give the application equal chance of acquiring
docker digest id on any node manager.
{quote}
I see. Why isn't the AM run inside of a Docker container?
The node might have an old Docker image on it. It would be nice to get the
image information from the registry and only fall back to the local node's
version if the registry lookup fails. An indirect way to do this would be to do
a {{docker pull} before calling {{docker images}}.
{quote}
If we don't care about the container and just want to know what the sha of the
image:tag is, then I agree with Chandni Singh that we don't need to use the
containerId.
Container ID is used by container executor to properly permission the working
directory, generate .cmd file for container-executor binary, and all output and
exit code are stored to the container id directory. Without container ID, we
will need to craft a complete separate path to acquire privileges to launch
docker commands, which is extra code duplication and not follow the security
practice that was baked in place to prevent parameter hijacking. I choose to
follow the existing process to avoid code bloat.
{quote}
If we can hit the docker registry directly via its REST API then we won't need
to invoke the container-executor at all and we can avoid this problem. This
looks like it should be fairly trivial, but I don't know how much more
difficult secure registries would be.
{quote}
But if there are many, couldn't that not be the correct one?
The output given from docker image [image-id] -f ".RepoDigests" may contain
similar names, like local/centos and centos at the same time due to fuzzy
matching. For loop matches the exact name instead of prefix matching. Hence, it
is always the correct one that is matched.
{quote}
Do you have documentation handy for {{docker image inspect}} that talks about
the fuzzy matching? The [docker
documentation|https://docs.docker.com/engine/reference/commandline/image_inspect/]
is pretty underwhelming. From my own testing, searching for a substring of an
image doesn't match.
{quote}
I think we should import this instead of including the full path
Sorry, can't do. There is another import that reference to
org.apache.hadoop.yarn.service.component.Component, which prevent use of the
same name.
{quote}
Ah yes. I figured there was a reason we didn't do this. Thanks for explaining.
{quote}
Spacing issues on the operators.
Checkstyle did not find spacing issue with the existing patch, and the issue is
not clear to me. Care to elaborate?
{quote}
{noformat}
+ if (compSpec.getArtifact()!=null && compSpec.getArtifact()
+ .getType()==TypeEnum.DOCKER) {
{noformat}
A space before and after {{!=}} and {{==}}. If the purpose of omitting the
spaces is to show operation bundling, then I would just add {{()}} around the
two separate comparisons around the {{&&}}
{quote}
The first part of both of these regexes is identical. I think we should create
a subregex and append to it to avoid having to make changes in multiple places
in the future. One if the image followed by a tag and the other is an image
followed by a sha. Should be easy to do.
Sure, I will compact this to rebase this patch to trunk.
{quote}
Thanks
{quote}
The else clause syntax doesn't seem to work for me. Did I do something wrong?
Yes, unlike C exec, when running docker command on cli, it needs to be quoted
to prevent shell expansion:
docker images --format="{{json .}}" --filter="dangling=flase"
. For clarity, we are using:
docker image [image-id] -f "{{.RepoDigests}}"
to find the real digest hash due to bugs with docker images output.
{quote}
Thanks for clearing up the quoting issue. But I'm still getting what appears to
be a less than ideal result. Is this expected behavior?
{noformat}
[ebadger@foo ~]$ sudo docker images --format="{{json .}}"
--filter=dangling=false
{}
{}
{}
{}
{}
{}
[ebadger@foo ~]$ docker -v
Docker version 1.13.1, build b2f74b2/1.13.1
{noformat}
{quote}
Another possible solution is to have the AM get the sha256 hash of the image
that it is running in and then passing that sha to all of the containers that
it starts. This would move the query into the Hadoop cluster itself.
I think the patch is implementing what you are suggesting that Hadoop query
into the cluster itself via a node manager rest endpoint.
{quote}
I see now. I didn't understand that ServiceScheduler was running as a part of
the native services AM. This makes sense.
Another question I have is around NMWebServices. Is this completely wide open
to anyone who can craft the correct endpoint or is there authentication done
before the {{docker images}} command will be run?
> Implement logic to keep docker image consistent in application that uses
> :latest tag
> ------------------------------------------------------------------------------------
>
> Key: YARN-9292
> URL: https://issues.apache.org/jira/browse/YARN-9292
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-9292.001.patch, YARN-9292.002.patch,
> YARN-9292.003.patch, YARN-9292.004.patch, YARN-9292.005.patch,
> YARN-9292.006.patch, YARN-9292.007.patch
>
>
> Docker image with latest tag can run in YARN cluster without any validation
> in node managers. If a image with latest tag is changed during containers
> launch. It might produce inconsistent results between nodes. This is surfaced
> toward end of development for YARN-9184 to keep docker image consistent
> within a job. One of the ideas to keep :latest tag consistent for a job, is
> to use docker image command to figure out the image id and use image id to
> propagate to rest of the container requests. There are some challenges to
> overcome:
> # The latest tag does not exist on the node where first container starts.
> The first container will need to download the latest image, and find image
> ID. This can introduce lag time for other containers to start.
> # If image id is used to start other container, container-executor may have
> problems to check if the image is coming from a trusted source. Both image
> name and ID must be supply through .cmd file to container-executor. However,
> hacker can supply incorrect image id and defeat container-executor security
> checks.
> If we can over come those challenges, it maybe possible to keep docker image
> consistent with one application.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
