[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16704227#comment-16704227 ]
Zhankun Tang edited comment on YARN-8714 at 11/30/18 6:47 AM: -------------------------------------------------------------- {quote}1) Why hardcoded to handle {{hdfs://}}, it could be s3, abfs, gs, etc. I'd definitely prefer to make this to more general. This including code and comments. {quote} {color:#d04437}Zhankun=>{color}Yeah. Agree with this. {quote}Because downloading files from remote fs could be risky. What if user accidentally specified "/"? If user has needs to localize a dir of file, he or she should tar or zip it before uploading to HDFS. {quote} {color:#d04437}Zhankun=>{color} Yeah. Maybe we could do capacity check to avoid risk. Let's confirm this requirment with [~liuxun323]. Is downloading from remote directory a good choice considring the potential security issue? was (Author: tangzhankun): {quote}1) Why hardcoded to handle {{hdfs://}}, it could be s3, abfs, gs, etc. I'd definitely prefer to make this to more general. This including code and comments. {quote} {color:#d04437}Zhankun=>{color}Yeah. Agree with this. {quote}Because downloading files from remote fs could be risky. What if user accidentally specified "/"? If user has needs to localize a dir of file, he or she should tar or zip it before uploading to HDFS. {quote} {color:#d04437}Zhankun=>{color} Yeah. Let's confirm this requirment with [~liuxun323]. Is downloading from remote directory a good choice considring the potential security issue? > [Submarine] Support files/tarballs to be localized for a training job. > ---------------------------------------------------------------------- > > Key: YARN-8714 > URL: https://issues.apache.org/jira/browse/YARN-8714 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Zhankun Tang > Priority: Major > Attachments: YARN-8714-WIP1-trunk-001.patch, > YARN-8714-WIP1-trunk-002.patch, YARN-8714-trunk.001.patch, > YARN-8714-trunk.002.patch > > > See > [https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7], > {{job run --localization ...}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org