[
https://issues.apache.org/jira/browse/YARN-9292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16803551#comment-16803551
]
Chandni Singh commented on YARN-9292:
-------------------------------------
{quote}
Real container id of the application master provides the already initialized
path and .cmd file is stored in existing container directory. cmd file gets
clean up when application is finished. Using randomly generated container id
will not clean up as nicely.
{quote}
[~eyang] In patch 6, a random container id is already being created on the
client side which is the {{ServiceScheduler}}. It is creating a container id
from the appId and the current system time.
{code}
+ ContainerId cid = ContainerId
+ .newContainerId(ApplicationAttemptId.newInstance(appId, 1),
+ System.currentTimeMillis());
{code}
For images, we probably need to write command file to a path independent of
containers under nmPrivate directory. Our code can ensure that once the
command is executed, the temp .cmd file is deleted.
I do think it is important that we don't expose this API with
container/container id in it because there is no logical relation between the
image and the container.
> Implement logic to keep docker image consistent in application that uses
> :latest tag
> ------------------------------------------------------------------------------------
>
> Key: YARN-9292
> URL: https://issues.apache.org/jira/browse/YARN-9292
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN-9292.001.patch, YARN-9292.002.patch,
> YARN-9292.003.patch, YARN-9292.004.patch, YARN-9292.005.patch,
> YARN-9292.006.patch
>
>
> Docker image with latest tag can run in YARN cluster without any validation
> in node managers. If a image with latest tag is changed during containers
> launch. It might produce inconsistent results between nodes. This is surfaced
> toward end of development for YARN-9184 to keep docker image consistent
> within a job. One of the ideas to keep :latest tag consistent for a job, is
> to use docker image command to figure out the image id and use image id to
> propagate to rest of the container requests. There are some challenges to
> overcome:
> # The latest tag does not exist on the node where first container starts.
> The first container will need to download the latest image, and find image
> ID. This can introduce lag time for other containers to start.
> # If image id is used to start other container, container-executor may have
> problems to check if the image is coming from a trusted source. Both image
> name and ID must be supply through .cmd file to container-executor. However,
> hacker can supply incorrect image id and defeat container-executor security
> checks.
> If we can over come those challenges, it maybe possible to keep docker image
> consistent with one application.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
