[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191311#comment-16191311 ] Jason Lowe commented on YARN-7226: -- The unit test failures appear to be unrelated, and they pass for me locally with the patch applied. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch, YARN-7226-branch-2.006.patch, > YARN-7226-branch-2.007.patch, YARN-7226-branch-2.8.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16191241#comment-16191241 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 47s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} branch-2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 183 unchanged - 5 fixed = 183 total (was 188) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 27m 19s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 16s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 42m 52s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.TestNodeManagerReboot | | Timed out junit tests | org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:eaf5c66 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890352/YARN-7226-branch-2.007.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux e118f3a17e00 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | branch-2 / 175ff05 | | Default Java | 1.7.0_151 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/17765/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17765/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17765/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190680#comment-16190680 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s{color} | {color:blue} Docker mode activated. {color} | | {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 8s{color} | {color:red} YARN-7226 does not apply to branch-2.8. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} | \\ \\ || Subsystem || Report/Notes || | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890178/YARN-7226-branch-2.8.006.patch | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17761/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch, YARN-7226-branch-2.006.patch, > YARN-7226-branch-2.8.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190674#comment-16190674 ] Sidharta Seethana commented on YARN-7226: - [~jlowe] I committed the branch-2.8 patch. Unfortunately, YARN-6550 was committed to branch-2 a couple of hours after you uploaded the patch and the branch-2 patch no longer applies. Apologies for the confusion - do you mind rebasing the branch-2 patch? > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch, YARN-7226-branch-2.006.patch, > YARN-7226-branch-2.8.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190265#comment-16190265 ] Jason Lowe commented on YARN-7226: -- The findbugs warning is unrelated. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch, YARN-7226-branch-2.006.patch, > YARN-7226-branch-2.8.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190260#comment-16190260 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2.8 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 32s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} branch-2.8 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 50s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} branch-2.8 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 22s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 92 unchanged - 5 fixed = 92 total (was 97) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 0s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 23m 47s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:c2d96dd | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890178/YARN-7226-branch-2.8.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux af4b47797001 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | branch-2.8 / 3629c95 | | Default Java | 1.7.0_151 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17752/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17752/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17752/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed variable expansion >
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16190214#comment-16190214 ] Jason Lowe commented on YARN-7226: -- The unit test failure is interesting. It's a latent bug caused by YARN-3069 that was masked by YARN-5444 in 2.9 and 3.x. I filed YARN-7285 to fix that bug properly, and I pulled YARN-5444 into branch-2.8 in the interim. I kicked Jenkins for another run. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch, YARN-7226-branch-2.006.patch, > YARN-7226-branch-2.8.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189941#comment-16189941 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 11m 29s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2.8 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 20s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 16s{color} | {color:green} branch-2.8 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} branch-2.8 passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 56s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} branch-2.8 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 90 unchanged - 5 fixed = 90 total (was 95) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 49s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 35m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:c2d96dd | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890178/YARN-7226-branch-2.8.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 6e1d1a8668dc 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | branch-2.8 / 4540cbb | | Default Java | 1.7.0_151 | | findbugs | v3.0.0 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17748/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/17748/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17748/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U:
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189866#comment-16189866 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} branch-2 Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 13s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 31s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 20s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 55s{color} | {color:green} branch-2 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} branch-2 passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 18s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 157 unchanged - 5 fixed = 157 total (was 162) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 14s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 30m 43s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 46m 0s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.webapp.TestNMWebServer | | | hadoop.yarn.server.nodemanager.TestNodeStatusUpdater | | | hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels | | | hadoop.yarn.server.nodemanager.TestNodeManagerReboot | | Timed out junit tests | org.apache.hadoop.yarn.server.nodemanager.TestNodeManagerResync | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:eaf5c66 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890166/YARN-7226-branch-2.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 4fd6582a7b1d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | branch-2 / 8beae14 | | Default Java | 1.7.0_151 | | findbugs | v3.0.0 | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/17745/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17745/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output |
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189200#comment-16189200 ] Hudson commented on YARN-7226: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13009 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13009/]) YARN-7226. Whitelisted variables do not support delayed variable (sidharta: rev 7eb846869cdb63743f1c9eca2ba91d57ad08) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/runtime/ContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DefaultLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16189191#comment-16189191 ] Sidharta Seethana commented on YARN-7226: - [~jlowe] +1 to the latest patch - committed to trunk and branch-3.0 . Could you please add a branch-2 version ? Thanks. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch, > YARN-7226.006.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188988#comment-16188988 ] Hadoop QA commented on YARN-7226: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 58s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 160 unchanged - 3 fixed = 160 total (was 163) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 12s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 59s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 56m 55s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890045/YARN-7226.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 671d74ce8d31 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2c62ff7 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17735/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17735/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188981#comment-16188981 ] Hadoop QA commented on YARN-7226: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 45s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 35s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 20s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 160 unchanged - 3 fixed = 160 total (was 163) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 15s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 54s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 15m 3s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 58m 22s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12890045/YARN-7226.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a4c510de8e65 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 2c62ff7 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17734/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17734/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16188820#comment-16188820 ] Sidharta Seethana commented on YARN-7226: - [~jlowe], recent changes to trunk (YARN-6550) seem to have modified {{ContainerLaunch.java}} and the latest patch from this JIRA no longer applies cleanly - could you please take a look? > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16186627#comment-16186627 ] Sidharta Seethana commented on YARN-7226: - I'll take a look at the latest patch today or Monday at the latest. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch, YARN-7226.005.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16184966#comment-16184966 ] Hadoop QA commented on YARN-7226: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 22s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 13m 54s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 54s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 130 unchanged - 3 fixed = 130 total (was 133) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 7s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 10s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 55m 0s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12889586/YARN-7226.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux f6d2adafebb7 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / ca669f9 | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17685/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17685/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181815#comment-16181815 ] Sidharta Seethana commented on YARN-7226: - [~jlowe], a couple of comments : * Is there a reason for adding {{useWhitelistEnv}} to {{LinuxContainerRuntime}} and not {{ContainerRuntime}} ? * Now that the whitelist behavior is different for the docker runtime, a test for this would be a good thing to add. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch, YARN-7226.004.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181699#comment-16181699 ] Hadoop QA commented on YARN-7226: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m 37s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 8m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 131 unchanged - 3 fixed = 131 total (was 134) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 1s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 31s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 52m 40s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12889161/YARN-7226.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux d861b90980da 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 0da29cb | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17648/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17648/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16181019#comment-16181019 ] Eric Badger commented on YARN-7226: --- Hey [~jlowe], thanks for the patch. I have one question on it. {noformat} + @Override + public boolean useWhitelistEnv(Mapenv) { +try { + LinuxContainerRuntime runtime = pickContainerRuntime(env); + return runtime.useWhitelistEnv(env); +} catch (ContainerExecutionException e) { + LOG.debug("Unable to determine runtime"); + return false; +} + } + {noformat} Is there a reason for us to catch the exception here? I would think that if we can't figure out which runtime to use that that would be fatal to the launch of the job. Only other thing would be to fix the whitespace checkstyle issue > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch, > YARN-7226.003.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179941#comment-16179941 ] Hadoop QA commented on YARN-7226: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 129 unchanged - 3 fixed = 130 total (was 132) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 7s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 17s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 16s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888961/YARN-7226.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 4cc913f74b36 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / cde804b | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17631/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17631/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17631/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL:
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179698#comment-16179698 ] Sidharta Seethana commented on YARN-7226: - {quote} But how does this work today for Docker containers? The launch script is using the var:-value syntax, which means if the image specified the value then it will not take the value the user desires. In other words, it looks like for Docker containers the semantics of the whitelist is the list of variables that cannot be overridden by the container. I hope we can all agree that if the container explicitly sets an environment variable then that variable should be set to the value the user specified. I think the only issue then is what to do about variables that are getting implicitly set via the NM whitelist that end up overriding the Docker image variables unintentionally. {quote} [~jlowe], you are right on both counts - short of moving more of the common functionality into the executors/container runtimes, there wasn't a better way of handling this at the time. The goal was to ensure that non-docker containers' behavior is not affected (however, based on this JIRA it looks like there was a scenario that wasn't considered). The approach of moving the implementation into the executors/runtimes makes sense to me as well - though keeping the semantics in sync (to the extent possible) is something we'll need to be careful about in the future. Maybe, at some point in the future, the docker implementation could use a different approach altogether (e.g specifying these env vars as a part of the docker run command). > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179676#comment-16179676 ] Eric Badger commented on YARN-7226: --- [~jlowe], this approach makes sense to me. I don't see any case where the docker container would want to use the NM's env vars over its own specified env vars, since the layout of the docker container is completely separate from that of the NM. So basically in the docker case, the docker container will either get what the user explicitly sets or whatever is in the docker image (in that order). And if we want to have variables that won't be overridden by anything (i.e. we take whatever the docker container sets no matter what), then we should do that in a different place than the whitelist and should file a followup JIRA. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179115#comment-16179115 ] Jason Lowe commented on YARN-7226: -- bq. If the user doesn't override the env vars, it is expected that what is specified in the image takes precedence. But how does this work today for Docker containers? The launch script is using the {{var:-value}} syntax, which means if the image specified the value then it will _not_ take the value the user desires. In other words, it looks like for Docker containers the semantics of the whitelist is the list of variables that _cannot_ be overridden by the container. I hope we can all agree that if the container explicitly sets an environment variable then that variable should be set to the value the user specified. I think the only issue then is what to do about variables that are getting implicitly set via the NM whitelist that end up overriding the Docker image variables unintentionally. Here's my proposal to fix this: The whitelist processing moves to the container runtime. * For non-Docker containers the behavior works semantically as in my patch (and before YARN-3853) where NM environment variables in the NM whitelist are added to the container's environment if and only if the user did not specify them. * For Docker containers the NM whitelist is *ignored*. Either the user set a value directly or it will implicitly pick it up from the Docker image. We won't try to port environment variables from the NM's env to the container env because we expect those variables to be specifically setup by the Docker image. Will that be acceptable? It will solve the case where JAVA_HOME is implicitly getting picked up from the NM whitelist and accidentally overriding the one in the Docker image. However if the user _explicitly_ sets JAVA_HOME then that user's value will be honored even if the Docker image specified it. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16177109#comment-16177109 ] Sidharta Seethana commented on YARN-7226: - [~ebadger], [~jlowe] {quote} The client gets to decide what image to use, so they should know the layout of the image anyway. And if they don't, then they can fall back to the environment of the docker image. {quote} {quote} I do see the problem where a container did not ask for any setting of some variable before entering the Docker container, but it gets a setting anyway because of NM whitelist processing (e.g.: HADOOP_COMMON_HOME or HADOOP_CONF_DIR). That whitelist-inherited setting can end up being problematic for the Docker image, e.g.: Hadoop is installed in a different place in the image compared to where the NM is picking it up. However I would counter-argue that since the user picked the Docker image, the user knows what values to set for these variables and can override them explicitly in the container startup. Basically I see it as a "would be nice to have" feature rather than "Docker containers do not work without it" feature. Am I missing the use case where it's really necessary? {quote} I believe this is not simply a 'nice to have' feature - the use case that does not work is exactly the one described above - a different HADOOP_COMMON_HOME or JAVA_HOME that is specified in the docker image is not picked up correctly. 'Software isolation' is the canonical use case for integrating docker containers with YARN - different java versions, hadoop versions etc., If the user doesn't override the env vars, it is expected that what is specified in the image takes precedence. If I understand correctly, 'delayed variable expansion' is broken (but other forms of env var override still work as expected for non-docker containers?) - is this expected behavior described somewhere ? > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176987#comment-16176987 ] Jason Lowe commented on YARN-7226: -- bq. So my vote would be for putting this patch in and then filing a followup jira to add a NM configurable list of variables that cannot be changed by the user (empty/disabled by default) I'm not sure I can come up with a great reason to suppress any variables for the non-Docker case. Besides, if a user really wants to get around that behavior they can just run a shell script as their command that in turn smashes the env vars the way they need it and runs the "real" command. IMHO this appears to call for a config specific to the Docker runtime that says "here are the list of variables I do not want to be overridden" or maybe relaxed to "here are the list of variables to preclude from the NM's whitelist". I do see the problem where a container did _not_ ask for any setting of some variable before entering the Docker container, but it gets a setting anyway because of NM whitelist processing (e.g.: HADOOP_COMMON_HOME or HADOOP_CONF_DIR). That whitelist-inherited setting can end up being problematic for the Docker image, e.g.: Hadoop is installed in a different place in the image compared to where the NM is picking it up. However I would counter-argue that since the user picked the Docker image, the user knows what values to set for these variables and can override them explicitly in the container startup. Basically I see it as a "would be nice to have" feature rather than "Docker containers do not work without it" feature. Am I missing the use case where it's really necessary? > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176961#comment-16176961 ] Eric Badger commented on YARN-7226: --- I can see the need for Docker containers to supply their own env vars in the image, but I agree that it should not change previous behavior of the whitelist. The client gets to decide what image to use, so they should know the layout of the image anyway. And if they don't, then they can fall back to the environment of the docker image. If we want for there to be a separate list of env vars that cannot be changed, then we can do that, but I think it should be empty and/or disabled by default to preserve current behavior. Otherwise, as [~jlowe] said, users won't be able to supply their own values for the env vars in the whitelist. So my vote would be for putting this patch in and then filing a followup jira to add a NM configurable list of variables that cannot be changed by the user (empty/disabled by default) > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16176376#comment-16176376 ] Jason Lowe commented on YARN-7226: -- bq. For example, an image used for running MR tasks could specify its own JAVA_HOME that needs to be used instead of the JAVA_HOME that is specified in the nodemanager's environment. But that's not what the whitelist is for. The whitelist is for the *nodemanager's environment variables* not the variable associated with the Docker image that the user requested to run with their job. That's not a nodemanager thing, that's a user's Docker image thing. In all cases, a whitelist should never prevent the container from overriding the variable, but that appears to be what this is doing for the Docker case. Any variable in the whitelist will not be set to the user's value _unless_ the Docker image doesn't specify it, correct? If so, that's going to be surprising to users. If Docker wants/needs the ability to "screen" certain variables from ever being set if they are specified in the image, whether that setting comes from the container context directly or the NM's environment via the whitelist, then IMHO that should be a separate config. Otherwise in a mixed-mode environment where some containers are running outside of Docker and some are running in Docker, how is an admin supposed to properly set yarn.nodemanager.env-whitelist properly? The set of variables in a Docker image and the set of NM variables that need to peek through to non-Docker containers may not be the same. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16175635#comment-16175635 ] Sidharta Seethana commented on YARN-7226: - {quote} As I understand it, a whitelisted variable is simply an environment variable that will be propagated from the nodemanager's environment to the container's environment unless the container has specified it's own value for that variable. {quote} [~jlowe], Using docker containers introduces a new way in which environment variables for a container can be specified - this is through the docker image itself, which is a fairly common scenario. For example, an image used for running MR tasks could specify its own JAVA_HOME that needs to be used instead of the JAVA_HOME that is specified in the nodemanager's environment. Since {{launch_container.sh}} runs inside the docker container, using the specified docker image, using {{putEnvIfAbsent}} for whitelisted env vars doesn't do the right thing in this case. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16174846#comment-16174846 ] Jason Lowe commented on YARN-7226: -- Pinging [~sidharta-s] and [~vvasudev] since the {{var=var:-value}} logic was added in YARN-3853 which broke this, and maybe I'm missing the use-case where this added logic is necessary. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173972#comment-16173972 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 19s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 125 unchanged - 2 fixed = 125 total (was 127) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 33s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 37m 51s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888120/YARN-7226.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux de2187d80afa 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a12f09b | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17553/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17553/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17553/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org | This message was automatically generated. > Whitelisted variables do not support delayed variable expansion >
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173930#comment-16173930 ] Jason Lowe commented on YARN-7226: -- Test failures appear to be unrelated, but unfortunately the details were lost and the links are broken. Kicking another Jenkins build for completeness. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > Attachments: YARN-7226.001.patch, YARN-7226.002.patch > > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173879#comment-16173879 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 33m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 17s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 52s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 1s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 52s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 0 new + 124 unchanged - 2 fixed = 124 total (was 126) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 24s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 55s{color} | {color:red} hadoop-yarn-server-nodemanager in the patch failed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m 15s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 18s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.server.nodemanager.TestNodeStatusUpdater | | | hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888120/YARN-7226.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 00ec1ec12642 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / a12f09b | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17547/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | unit | https://builds.apache.org/job/PreCommit-YARN-Build/17547/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17547/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U:
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173478#comment-16173478 ] Hadoop QA commented on YARN-7226: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 59s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m 29s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green} trunk passed {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 51s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 22s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 2 new + 124 unchanged - 2 fixed = 126 total (was 126) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 14m 58s{color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 23s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 53m 58s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:71bbb86 | | JIRA Issue | YARN-7226 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12888071/YARN-7226.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 268bd87f2664 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 230b85d | | Default Java | 1.8.0_144 | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/PreCommit-YARN-Build/17539/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/17539/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/17539/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/17539/console | | Powered by | Apache Yetus 0.6.0-SNAPSHOT
[jira] [Commented] (YARN-7226) Whitelisted variables do not support delayed variable expansion
[ https://issues.apache.org/jira/browse/YARN-7226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16173232#comment-16173232 ] Jason Lowe commented on YARN-7226: -- Maybe I'm missing something, but the whitelisted variable support seems to be a lot more complicated than it needs to be. As I understand it, a whitelisted variable is simply an environment variable that will be propagated from the nodemanager's environment to the container's environment _unless_ the container has specified it's own value for that variable. Rather than have special whitelisted support when writing the shell scripts, etc., it seems all we need to do is: # Set all the environment variables in the env variable map that the container specified as we do today. # Do a subsequent pass over the whitelisted variables and simply leverage {{putEnvIfAbsent}} to propagate the variable into the container's env if it wasn't set in the previous step ...and that's it. No special script handling of these variables and no need for {{var:-default}} variable expansion syntax. The latter causes the problem because the value is always set in the environment of the launch_container.sh process before the script executes, so it never takes the default value. > Whitelisted variables do not support delayed variable expansion > --- > > Key: YARN-7226 > URL: https://issues.apache.org/jira/browse/YARN-7226 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.9.0, 2.8.1, 3.0.0-alpha4 >Reporter: Jason Lowe >Assignee: Jason Lowe > > The nodemanager supports a configurable list of environment variables, via > yarn.nodemanager.env-whitelist, that will be propagated to the container's > environment unless those variables were specified in the container launch > context. Unfortunately the handling of these whitelisted variables prevents > using delayed variable expansion. For example, if a user shipped their own > version of hadoop with their job via the distributed cache and specified: > {noformat} > HADOOP_COMMON_HOME={{PWD}}/my-private-hadoop/ > {noformat} > as part of their job, the variable will be set as the *literal* string: > {noformat} > $PWD/my-private-hadoop/ > {noformat} > rather than having $PWD expand to the container's current directory as it > does for any other, non-whitelisted variable being set to the same value. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org