[ 
https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508689#comment-15508689
 ] 

Jian He edited comment on YARN-5621 at 9/21/16 4:40 AM:
--------------------------------------------------------

Thanks for your explanation,  some parts I'm missing: 
bq. Both will start ContainerLocalizer instances
Is it starting both instances now?  Not sure if I read the code wrong...  It 
seems not the case. Based on the code, if it's an already existing resource,  
it will NOT start the ContainerLocalizer. e.g. below comments in 
ResourceLocalizationService.  Could you point me the right cod... maybe I'm 
missing something.
{code}
         /*
          * Multiple containers will try to download the same resource. So the
          * resource download should start only if
          * 1) We can acquire a non blocking semaphore lock on resource
          * 2) Resource is still in DOWNLOADING state
          */
{code}
bq.  it may be used by both, concurrently.
This approach may not be easily worked for the new containers without 
structural change, because for new containers, when localizer is started, the 
work-dirs are not setup yet. It cannot create symlinks on localization for new 
containers. The work-dirs are created later when launching the container.
bq.  for services that upgrade over minutes/hours,
Not only for upgrades, it's also used by Tez for localizing resources on 
demand. 

I think I understand your approach now,  basically,
To create the symlinks,
1. we start the localizer process,
2. send the symlinks over on localizer heartbeat, 
3. localizer process create symlinks. 
right ?


was (Author: jianhe):
Thanks for your explanation,  some parts I'm missing: 
bq. Both will start ContainerLocalizer instances
Is it starting both instances now?  Not sure if I read the code wrong...  It 
seems not the case. Based on the code, if it's an already existing resource,  
it will NOT start the ContainerLocalizer. e.g. below comments in 
ResourceLocalizationService.  Could you point me the right cod... maybe I'm 
missing something.
{code}
         /*
          * Multiple containers will try to download the same resource. So the
          * resource download should start only if
          * 1) We can acquire a non blocking semaphore lock on resource
          * 2) Resource is still in DOWNLOADING state
          */
{code}
bq.  it may be used by both, concurrently.
This approach may not be easily worked for the new containers without 
structural change, because for new containers, when localizer is started, the 
work-dirs are not setup yet. It cannot create symlinks on localization for new 
containers. The work-dirs are created later when launching the container.
bq.  for services that upgrade over minutes/hours,
Not only for upgrades, it's also used by Tez for localizing resources on 
demand. 

I think I understand your approach now,  basically,
To create the symlinks,
1. we start the localizer process,
2. send the symlinks over on localizer heartbeat, 
3. localizer process create symlinks. 

One question is, should we keep the localizer process around or terminates it 
immediately after symlinks created. If we keep it around for certain time, we 
need to add some life timeout for the localizer process. If it's immediately 
terminated,  that means every localize request by AM would spawn/kill a new 
process, and I think that could churn over the machine resources.

> Support LinuxContainerExecutor to create symlinks for continuously localized 
> resources
> --------------------------------------------------------------------------------------
>
>                 Key: YARN-5621
>                 URL: https://issues.apache.org/jira/browse/YARN-5621
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Jian He
>            Assignee: Jian He
>         Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch, 
> YARN-5621.4.patch, YARN-5621.5.patch
>
>
> When new resources are localized, new symlink needs to be created for the 
> localized resource. This is the change for the LinuxContainerExecutor to 
> create the symlinks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org

Reply via email to