[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16464056#comment-16464056 ] Billie Rinaldi commented on YARN-8079: -- +1 for patch 10. It seems like we have all agreed, so I will go ahead and commit this patch. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch, YARN-8079.010.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463279#comment-16463279 ] genericqa commented on YARN-8079: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 2 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 34s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 52s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 14s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 56s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 27s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 85m 44s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-8079 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12921869/YARN-8079.010.patch | | Optional Tests | asflicense compile javac javadoc mv
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16463238#comment-16463238 ] Suma Shivaprasad commented on YARN-8079: Thanks [~billie.rinaldi] [~gsaha] [~eyang] for your suggestions. Discussed offline with [~billie.rinaldi] [~gsaha] [~eyang] , STATIC seems to be an agreed upon name since the contents of the file are not modified by YARN service and is also a known name in the web terminology to indicate that static files are served as is without modification on the server. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch, YARN-8079.010.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462973#comment-16462973 ] Gour Saha commented on YARN-8079: - Or maybe call it SIMPLE > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462928#comment-16462928 ] Eric Yang commented on YARN-8079: - [~gsaha] PLAIN is usually used as reference to text file, but this feature applies to files in general. I am ok to go with FILE, if most people don't find "files": \{ "type": "FILE", ...} wordy or confusing. Maybe we default type to FILE, if "type" has not be explicitly defined. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462876#comment-16462876 ] Gour Saha commented on YARN-8079: - I agree that STATIC is not fitting well. I like FILE. Does PLAIN fit the profile better? [~suma.shivaprasad] few minor comments - 1. Please remove the unnecessary import and new line changes in Component.java and TestYarnNativeServices.java. 2. In the description of type for ConfigFile in YarnServiceAPI.md make "STATIC/ARCHIVE" lowercase to be in-line with the others. Also put a period at the end of the sentence. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462872#comment-16462872 ] Eric Yang commented on YARN-8079: - [~billie.rinaldi] STATIC works better in my opinion. It helps to differentiate between template file and static file. It also helps to have a slight different word from files because the upper level closure is already using keyword "files". > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16462859#comment-16462859 ] Billie Rinaldi commented on YARN-8079: -- I tested patch 9 and it is working fine, so I think this is about ready for commit. One last question for everyone, is STATIC a good name for the file type that is localized without changes, or should we consider a different name (e.g. FILE to match the local resource type)? > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461682#comment-16461682 ] genericqa commented on YARN-8079: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 35s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 39s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 36s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 27s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 49s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-8079 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12921631/YARN-8079.009.patch | | Optional Tests | asflicense compile javac javadoc mv
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16461589#comment-16461589 ] Suma Shivaprasad commented on YARN-8079: Fixed remaining checkstyle issues > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch, > YARN-8079.009.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460121#comment-16460121 ] genericqa commented on YARN-8079: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 28s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 25s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 6s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 7s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 7s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 59 unchanged - 0 fixed = 61 total (was 59) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 14s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 33s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 21s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 21s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 90m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-8079 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachmen
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16460017#comment-16460017 ] Suma Shivaprasad commented on YARN-8079: Thanks [~billie.rinaldi] Uploaded Patch with review comments addressed > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459698#comment-16459698 ] Billie Rinaldi commented on YARN-8079: -- Hi [~suma.shivaprasad], thanks for picking this up. The patch is looking good overall; I only have a couple of small comments. It looks like we aren't checking whether srcFile is a directory in AbstractClientProvider (we're only checking whether the file exists). It would be good to add that check to AbstractClientProvider so that it is being validated at app submission time. The comments added to createConfigFileAndAddLocalResource are not quite accurate. It says: "When source file not specified, upload new configs to compInstanceDir/fileName. Otherwise, use sourceFile." and "Output properties to sourceFile if not existed." The method always writes the configs to compInstanceDir/fileName. When srcFile isn't specified, it writes new configs. When srcFile is specified, it reads the srcFile, performs variable substitution and merges in new configs, and writes a new file to compInstanceDir/fileName. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459388#comment-16459388 ] genericqa commented on YARN-8079: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 3 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 2s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 26s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 7s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 16s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 27s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 22s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 14s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 42s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 42s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 13s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 4 new + 89 unchanged - 0 fixed = 93 total (was 89) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 20s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 26s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 57s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 47s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 37s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}113m 16s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | had
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16459328#comment-16459328 ] Suma Shivaprasad commented on YARN-8079: Uploaded patch after testing with both STATIC/ARCHIVE types with the following changes to the patch # Modified TestAbstractClientProvider to test validation changes # Changed localization destination from "localized" to "resources Service spec to test the above {code} { "name": "sleeper-service", "version": "1", "components": [{ "name": "sleep", "number_of_containers": 2, "launch_command": "sleep 9", "resource": { "cpus": 1, "memory": "256" }, "configuration": { "files": [{ "type": "STATIC", "dest_file": "sometextfile", "src_file": "hdfs:///tmp/a.json" }] } }] } {code} > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch, YARN-8079.007.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455609#comment-16455609 ] Wangda Tan commented on YARN-8079: -- I'm a bit packed recently to finish this patch, discussed with [~suma.shivaprasad] offline and she will offer help to finish this JIRA. Just reassigned. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Suma Shivaprasad >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455562#comment-16455562 ] Wangda Tan commented on YARN-8079: -- Found previous spec mentioned has some issues, for the files part, it should be: {code} "components" : [ { "name" : "primary-worker", "configuration" : { "properties" : { }, "files" : [ { "type" : "STATIC", "dest_file" : "run-PRIMARY_WORKER.sh", "src_file" : "hdfs://:8020/file", } ] }, } ] {code} > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455545#comment-16455545 ] genericqa commented on YARN-8079: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color} | {color:red} YARN-8079 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-8079 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12920908/YARN-8079.006.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/20502/console | | Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455544#comment-16455544 ] Suma Shivaprasad commented on YARN-8079: [~leftnoteasy] As discussed offline, I can take up the remaining effort on this patch. Thanks > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8079) Support specify files to be downloaded (localized) before containers launched by YARN
[ https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16455541#comment-16455541 ] Wangda Tan commented on YARN-8079: -- Attached ver.6 patch, fixed all issues. Spec mentioned by Eric above: https://issues.apache.org/jira/browse/YARN-8079?focusedCommentId=16419665&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16419665 could be used to do tests. > Support specify files to be downloaded (localized) before containers launched > by YARN > - > > Key: YARN-8079 > URL: https://issues.apache.org/jira/browse/YARN-8079 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Critical > Attachments: YARN-8079.001.patch, YARN-8079.002.patch, > YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch, > YARN-8079.006.patch > > > Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly > read srcFile, instead it always construct {{remoteFile}} by using > componentDir and fileName of {{destFile}}: > {code} > Path remoteFile = new Path(compInstanceDir, fileName); > {code} > To me it is a common use case which services have some files existed in HDFS > and need to be localized when components get launched. (For example, if we > want to serve a Tensorflow model, we need to localize Tensorflow model > (typically not huge, less than GB) to local disk. Otherwise launched docker > container has to access HDFS. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org