[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692696#comment-16692696
]
Zhankun Tang edited comment on YARN-8714 at 11/20/18 5:55 AM:
--------------------------------------------------------------
[~liuxun323], [~leftnoteasy], [~yuan_zac]. Updated a new version. More tests
are in progress. But please help to review in case our direction is not right.
For all ps and worker component, it provides --localization to support both
HDFS and local directory (may download, then zip it in temp dir and then upload
to HDFS) as the first part of the parameter. And it also supports mount into
the container with permission(default is rw)
Goals are:
{code:java}
--localization hdfs:///user/yarn/mydir:.:ro # download, zip and upload the zip
file to HDFS. then mount into container word dir's "mydir" folder. {PWD}/mydir
--localization /user/yarn/mydir2:/opt/dir2 # zip local dir and upload the zip
file to HDFS. set mount into container's /opt/dir2
--localization /user/yarn/script1.py:.:ro # upload a script to HDFS and set
mount into container {PWD}/script1.py
--localization /user/yarn/script1.py:/opt/script2.py:rw # upload and mount into
container /opt/script2.py
{code}
was (Author: tangzhankun):
[~liuxun323], [~leftnoteasy], [~yuan_zac]. Updated a new version. More tests
are needed. But please help to review incase the changes are too big.
For all ps and worker component, it provides --localization to support both
HDFS and local directory (may download, then zip it in temp dir and then upload
to HDFS) as the first part of the parameter. And it also supports mount into
the container with permission(default is rw)
Goals are:
{code:java}
--localization hdfs:///user/yarn/mydir:.:ro # download, zip and upload the zip
file to HDFS. then mount into container word dir's "mydir" folder. {PWD}/mydir
--localization /user/yarn/mydir2:/opt/dir2 # zip local dir and upload the zip
file to HDFS. set mount into container's /opt/dir2
--localization /user/yarn/script1.py:.:ro # upload a script to HDFS and set
mount into container {PWD}/script1.py
--localization /user/yarn/script1.py:/opt/script2.py:rw # upload and mount into
container /opt/script2.py
{code}
> [Submarine] Support files/tarballs to be localized for a training job.
> ----------------------------------------------------------------------
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Zhankun Tang
> Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch
>
>
> See
> https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7,
> {{job run --localizations ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
