[
https://issues.apache.org/jira/browse/YARN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14347583#comment-14347583
]
Chen He commented on YARN-3289:
-------------------------------
Thank you for the quick feedback, [~jlowe].
{quote} What we're missing here is progress reporting during localization so
AMs can properly monitor progress of container launch requests before their
code starts running, and that's useful for non-docker localization scenarios as
well.{quote}
I agree. That will be great. The idea that I proposed is based on the condition
that we do not chance localization part.
{quote} One node may take tens of minutes to localize a docker image, but
another node might only take a few seconds. Docker images are often derived
from other images, and docker only downloads the deltas. So it will be
difficult for YARN that is not aware of the docker contents of a node or image
deltas to predict how long any node will take to localize a given docker image.
So it will be difficult for YARN that is not aware of the docker contents of a
node or image deltas to predict how long any node will take to localize a given
docker image.{quote}
That is true. Docker image localization is a little bit different from other
APP localization process (from HDFS to localFS). They all pull from docker
registry. The network bandwidth from docker registry to each NM could be a
bottleneck no matter whether the docker image deltas is large or small (we may
need higher bandwidth, let's say 30G infi-band. But for a larger Hadoop
cluster, more than 10 thousand task running, it may still be a problem). This
is another reason that we need to consider docker image locality.
> Docker images should be downloaded during localization
> ------------------------------------------------------
>
> Key: YARN-3289
> URL: https://issues.apache.org/jira/browse/YARN-3289
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Ravi Prakash
>
> We currently call docker run on images while launching containers. If the
> image size if sufficiently big, the task will timeout. We should download the
> image we want to run during localization (if possible) to prevent this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
