[
https://issues.apache.org/jira/browse/YARN-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475973#comment-15475973
]
Jian He edited comment on YARN-5621 at 9/9/16 5:32 AM:
-------------------------------------------------------
bq. the container launch script is pretty self contained, is mostly controlled
by the NM, and there are other actions in the pipeline. Running a generic
script without any of that extra baggage around it seems to be greatly
expanding the footprint of c-e.
I still don't see the difference. Both are controlled by NM. The
launch_container.sh can run any arbitrary user supplied command. If you just
pass dummy objects to other parameters of container launch command, it's
essentially doing exactly the same thing.
bq. Why can't the directory structure be built by the NM and it just be a chown
operation by c-e?
I guess the question is why the original container_launch script is not done in
this way? Even if it's done this way, should we then add a new 'chown' API in
the container-executor ? or should it belong to the API of creating symlinks
(similar as a script)? It's much easier to follow existing container_launch
code which takes care of all environments, rather than inventing new. Also,
later on we need to create multiple symlinks in a single operation as done in
current container_launch script, because if there is a large number of local
Resources to be localized, we don't want to invoke the binary for each of them.
was (Author: jianhe):
bq. the container launch script is pretty self contained, is mostly controlled
by the NM, and there are other actions in the pipeline. Running a generic
script without any of that extra baggage around it seems to be greatly
expanding the footprint of c-e.
I still don't see the difference. Both are controlled by NM. The
launch_container.sh runs any arbitrary user supplied command and this feature
does the same thing. If you just pass dummy objects to other parameters of
container launch command, it's essentially doing exactly the same thing.
bq. Why can't the directory structure be built by the NM and it just be a chown
operation by c-e?
I guess the question is why the original container_launch script is not done in
this way? Even if it's done this way, should we then add a new 'chown' API in
the container-executor ? or should it belong to the API of creating symlinks
(similar as a script)? It's much easier to follow existing container_launch
code which takes care of all environments, rather than inventing new. Also,
later on we need to create multiple symlinks in a single operation as done in
current container_launch script, because if there is a large number of local
Resources to be localized, we don't want to invoke the binary for each of them.
> Support LinuxContainerExecutor to create symlinks for continuously localized
> resources
> --------------------------------------------------------------------------------------
>
> Key: YARN-5621
> URL: https://issues.apache.org/jira/browse/YARN-5621
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Jian He
> Assignee: Jian He
> Attachments: YARN-5621.1.patch, YARN-5621.2.patch, YARN-5621.3.patch
>
>
> When new resources are localized, new symlink needs to be created for the
> localized resource. This is the change for the LinuxContainerExecutor to
> create the symlinks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
