[
https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16709547#comment-16709547
]
Zhankun Tang edited comment on YARN-8714 at 12/5/18 2:33 AM:
-------------------------------------------------------------
[~leftnoteasy], [~liuxun323] . Per my testing, the YARN localizer can handle
directory. I changed distributedShell's client to let it localize an HDFS
directory "mydir" directly.
{code:java}
Path p = new Path("hdfs:///user/yarn/submarine/jobs/tf-job-001/staging" +
"/mydir");
FileStatus scFileStatus = fs.getFileStatus(p);
LocalResource r =
LocalResource.newInstance(URL.fromURI(p.toUri()),
LocalResourceType.FILE, LocalResourceVisibility.APPLICATION,
scFileStatus.getLen(), scFileStatus.getModificationTime());
localResources.put("mydir", r);{code}
And YARN localizer indeed downloads the HDFS dir to local for DistributedShell.
{code:java}
yarn@master0-VirtualBox:~/apache-hadoop-install-dir/hadoop-dev-workspace$ ls
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/13/mydir
1.py 2.py dir1 test_kill9.sh
yarn@master0-VirtualBox:~/apache-hadoop-install-dir/hadoop-dev-workspace$ ls
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/container_1543898305524_0004_01_000001/
-l
total 20
lrwxrwxrwx 1 yarn hadoop 111 12月 5 10:08 AppMaster.jar ->
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/10/AppMaster.jar
lrwxrwxrwx 1 yarn hadoop 103 12月 5 10:08 mydir ->
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/13/mydir
{code}
But the bad news is submarine utilizes YARN native service, and it doesn't know
this YARN ability and blocked it.
{code:java}
2018-12-05 10:06:40,286 ERROR client.ApiServiceClient:
srcFile=hdfs://192.168.50.191:9000/user/yarn/submarine/jobs/tf-job-001/staging/mydir
is a directory, which is not supported.
{code}
Two solutions ahead of us at present:
1. Fix the improper handling of the directory in the native service and then
got this implement.
2. Go ahead with our download, zip and upload approach which is more complex.
And refactor this after 1 is done.
Any thoughts?
was (Author: tangzhankun):
[~leftnoteasy], [~liuxun323] . Per my testing, the YARN localizer can handle
directory. I changed distributedShell's client to let it localize an HDFS
directory "mydir" directly.
{code:java}
Path p = new Path("hdfs:///user/yarn/submarine/jobs/tf-job-001/staging" +
"/mydir");
FileStatus scFileStatus = fs.getFileStatus(p);
LocalResource r =
LocalResource.newInstance(URL.fromURI(p.toUri()),
LocalResourceType.FILE, LocalResourceVisibility.APPLICATION,
scFileStatus.getLen(), scFileStatus.getModificationTime());
localResources.put("mydir", r);{code}
And YARN localizer indeed downloads the HDFS dir to local for DistributedShell.
{code:java}
yarn@master0-VirtualBox:~/apache-hadoop-install-dir/hadoop-dev-workspace$ ls
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/13/mydir
1.py 2.py dir1 test_kill9.sh
yarn@master0-VirtualBox:~/apache-hadoop-install-dir/hadoop-dev-workspace$ ls
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/container_1543898305524_0004_01_000001/
-l
total 20
lrwxrwxrwx 1 yarn hadoop 111 12月 5 10:08 AppMaster.jar ->
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/10/AppMaster.jar
lrwxrwxrwx 1 yarn hadoop 103 12月 5 10:08 mydir ->
/tmp/hadoop-yarn/nm-local-dir/usercache/yarn/appcache/application_1543898305524_0004/filecache/13/mydir
{code}
But the bad news is submarine utilizes YARN native server, and it doesn't know
this YARN ability and blocked it.
{code:java}
2018-12-05 10:06:40,286 ERROR client.ApiServiceClient:
srcFile=hdfs://192.168.50.191:9000/user/yarn/submarine/jobs/tf-job-001/staging/mydir
is a directory, which is not supported.
{code}
Two solutions ahead of us at present:
1. Fix the improper handling of the directory in the native service and then
got this implement.
2. Go ahead with our download, zip and upload approach which is more complex.
And refactor this after 1 is done.
Any thoughts?
> [Submarine] Support files/tarballs to be localized for a training job.
> ----------------------------------------------------------------------
>
> Key: YARN-8714
> URL: https://issues.apache.org/jira/browse/YARN-8714
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Assignee: Zhankun Tang
> Priority: Major
> Attachments: YARN-8714-WIP1-trunk-001.patch,
> YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch,
> YARN-8714-trunk.002.patch
>
>
> See
> [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7],
> {{job run --localization ...}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
