Chris Douglas commented on YARN-5621:

bq. I think I understand your approach now, basically, [...]

Yes, that's the gist of it. The {{ContainerLocalizer}} manages the private 
cache as the user, with that user's cluster credentials. Running containers 
start a CL to download private resources and/or create symlinks.

bq. Is it starting both instances now? Not sure if I read the code wrong... It 
seems not the case. Based on the code, if it's an already existing resource, it 
will NOT start the ContainerLocalizer.

[container start] For different applications? It should. For container start, I 
don't remember offhand if the {{ContainerLocalizer}} spawn is delayed until at 
least one dependent resource is not claimed, but IIRC it starts if at least one 
resource is not downloaded. Either way, CLs could start in race for a resource 
_R_, and only one would (successfully) download it. Resources aren't claimed 
when the CL launches, only when it heartbeats in.

[CL proposal] For running container localization and for rollback, the CL will 
download the resource (again) and/or create the symlink to the running 
container. If multiple containers/applications request the same resource, it 
doesn't matter if it's a mix of new/running containers requesting a resource 
_R_. Only running/rollback containers will send symlink commands to their CL.

bq. This approach may not be easily worked for the new containers without 
structural change, when localizer is started, the work-dirs are not setup yet.

Again, container start is unaffected; new containers will not send {{LINK}} 
commands to the CL. Only _running_ containers will start a CL that receives 
{{LINK}} commands, after the work dirs are created and the container has 

> Support LinuxContainerExecutor to create symlinks for continuously localized 
> resources
> --------------------------------------------------------------------------------------
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, 
> YARN-5621.4.patch, YARN-5621.5.patch
> When new resources are localized, new symlink needs to be created for the 
> localized resource. This is the change for the LinuxContainerExecutor to 
> create the symlinks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to