[
https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15522339#comment-15522339
]
Jian He commented on YARN-5621:
-------------------------------
I still have hesitation on this approach to spawn an additional process to only
create symlinks, as it is heavier than directly invoking container-executor and
implementation is more complicated.
I checked the code, it seems not that simple to make the whole flow work.
Concretely ,
- When container creates the symlink, containerImpl needs to send a new LINK
event to ResourceLocalizationService which will create a new LocalizerRunner
thread- a number of refactors need to be made here so that all parameters are
made available to the LocalizerRunner.
- Then LocalizerRunner#findNextResource internally the logic need to be forked
to send over symlinks instead of resources.
- LocalizerHeatBeatRequest: One LINK object with symlink payloads need to be
added in the heartBeat request,
- LocalizerHeatBeatResponse: Two more objects LINK_FAILED and LINK_SUCCEED need
to be added, and also the resource the link corresponds to.
- Once done, ResourceLocalizationService need to send LINK_FAILED and
LINK_SUCCEED events back to ContainerImpl
- ContainerImpl state-machine gets complicated mainly because the symlink
operation becomes asynchronous.
-- first it needs to keep track of all original LINK requests so that when
ResourceLocalizationService sends the results back, it knows which resource
this request was originally correspond to. User needs to query the status.
-- new state-transitions also need to be added to handle the new LINK_FAILED
and LINK_SUCCEEDED events.
I also would like to know what changes I need to make if we choose the
container-executor approach.
[~vvasudev], what's your opinion ?
> Support LinuxContainerExecutor to create symlinks for continuously localized
> resources
> --------------------------------------------------------------------------------------
>
> Key: YARN-5621
> URL: https://issues.apache.org/jira/browse/YARN-5621
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Jian He
> Assignee: Jian He
> Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch,
> YARN-5621.4.patch, YARN-5621.5.patch
>
>
> When new resources are localized, new symlink needs to be created for the
> localized resource. This is the change for the LinuxContainerExecutor to
> create the symlinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
