[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16708874#comment-16708874 ]
Zhankun Tang edited comment on YARN-8714 at 12/4/18 3:32 PM: ------------------------------------------------------------- [~leftnoteasy], [~liuxun323] , While refining the patch, I found that YARN localizer seems can localize *remote directory(hdfs, s3 .etc)*. In FSDownload.java#downloadAndUnpack, it uses "FileUtil.copy" which can handle directory. Depending on this can greatly simplify our implementation, no need to download remote dir or zip local dir anymore. We may still need a configuration to limit the remote file/dir size to be localized to the container. I will verify and update the patch tomorrow. was (Author: tangzhankun): [~leftnoteasy], [~liuxun323] , While refining the patch, I found that YARN localizer seems can localize *remote directory(hdfs, s3 .etc)*. In FSDownload.java#downloadAndUnpack, it uses "FileUtil.copy" which can handle directory. Depending on this can greatly simplify our implementation, no need to download remote dir or zip local dir anymore. I will verify and update the patch tomorrow. > [Submarine] Support files/tarballs to be localized for a training job. > ---------------------------------------------------------------------- > > Key: YARN-8714 > URL: https://issues.apache.org/jira/browse/YARN-8714 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Zhankun Tang > Priority: Major > Attachments: YARN-8714-WIP1-trunk-001.patch, > YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch, > YARN-8714-trunk.002.patch > > > See > [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7], > {{job run --localization ...}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org