[
https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15510866#comment-15510866
]
Chris Douglas commented on YARN-5621:
-------------------------------------
bq. I think I understand your approach now, basically, [...]
Yes, that's the gist of it. The {{ContainerLocalizer}} manages the private
cache as the user, with that user's cluster credentials. Running containers
start a CL to download private resources and/or create symlinks.
bq. Is it starting both instances now? Not sure if I read the code wrong... It
seems not the case. Based on the code, if it's an already existing resource, it
will NOT start the ContainerLocalizer.
[container start] For different applications? It should. For container start, I
don't remember offhand if the {{ContainerLocalizer}} spawn is delayed until at
least one dependent resource is not claimed, but IIRC it starts if at least one
resource is not downloaded. Either way, CLs could start in race for a resource
_R_, and only one would (successfully) download it. Resources aren't claimed
when the CL launches, only when it heartbeats in.
[CL proposal] For running container localization and for rollback, the CL will
download the resource (again) and/or create the symlink to the running
container. If multiple containers/applications request the same resource, it
doesn't matter if it's a mix of new/running containers requesting a resource
_R_. Only running/rollback containers will send symlink commands to their CL.
bq. This approach may not be easily worked for the new containers without
structural change, when localizer is started, the work-dirs are not setup yet.
Again, container start is unaffected; new containers will not send {{LINK}}
commands to the CL. Only _running_ containers will start a CL that receives
{{LINK}} commands, after the work dirs are created and the container has
started.
> Support LinuxContainerExecutor to create symlinks for continuously localized
> resources
> --------------------------------------------------------------------------------------
>
> Key: YARN-5621
> URL: https://issues.apache.org/jira/browse/YARN-5621
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Jian He
> Assignee: Jian He
> Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch,
> YARN-5621.4.patch, YARN-5621.5.patch
>
>
> When new resources are localized, new symlink needs to be created for the
> localized resource. This is the change for the LinuxContainerExecutor to
> create the symlinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
