[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15612952#comment-15612952 ] Arun Suresh commented on YARN-3755: --- The patch does not seem to apply cleanly to branch-2.7. Can you please update it Jeff ? > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Labels: oct16-easy > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15425766#comment-15425766 ] Hadoop QA commented on YARN-3755: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s {color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 27s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 40s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s {color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 14s {color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: The patch generated 1 new + 59 unchanged - 1 fixed = 60 total (was 60) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 25s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 26s {color} | {color:green} hadoop-yarn-server-nodemanager in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 15s {color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 26m 48s {color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Image:yetus/hadoop:9560f25 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12736781/YARN-3755-2.patch | | JIRA Issue | YARN-3755 | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle | | uname | Linux 1f1b6dc18b15 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh | | git revision | trunk / 20f0eb8 | | Default Java | 1.8.0_101 | | findbugs | v3.0.0 | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/12817/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/12817/testReport/ | | modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/12817/console | | Powered by | Apache Yetus 0.3.0 http://yetus.apache.org | This message was automatically generated. > Log the command of launching containers > ---
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577097#comment-14577097 ] Steve Loughran commented on YARN-3755: -- I think this should be re-opened with YARN-3759 tagged as a duplicate. # one of the problems of pushing it to the framework is "what if you can't control that framework"? Because I spent a lot of time last week having to add the logging inside my own fork of spark do to an "env > $LOG_DIR/env" # your framework code doesn't know the CLI and env that is finally generated I propose, then, one of two options # that YARN has the option to log the env and CLI to the log directory # that we have a specific YARN log for app & container launch, which can be set to INFO to get this log information # that we start a section in the docs "troubleshooting YARN" which explains how to use this stuff. This patch would be basis the no. 2 item on that list; we just create a specific log for those messages. > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570276#comment-14570276 ] Jeff Zhang commented on YARN-3755: -- Close it as won't fix > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14570275#comment-14570275 ] Jeff Zhang commented on YARN-3755: -- bq. How about we let individual frameworks like MapReduce/Tez log them as needed? That seems like the right place for debugging too - app developers don't always get access to the daemon logs. Make sense. > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14569347#comment-14569347 ] Vinod Kumar Vavilapalli commented on YARN-3755: --- We had this long ago in YARN, but removed it as the log files were getting inundated in large/high throughput clusters. If you combine the command line with the environment (classpath etc), this can get very long. How about we let individual frameworks like MapReduce/Tez log them as needed? That seems like the right place for debugging too - app developers don't always get access to the daemon logs. > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568801#comment-14568801 ] Hadoop QA commented on YARN-3755: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 15m 45s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | | {color:green}+1{color} | javac | 7m 38s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 33s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 36s | The applied patch generated 1 new checkstyle issues (total was 58, now 58). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 35s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | yarn tests | 6m 7s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 43m 31s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12736781/YARN-3755-2.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 990078b | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8165/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8165/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8165/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8165/console | This message was automatically generated. > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch, YARN-3755-2.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3755) Log the command of launching containers
[ https://issues.apache.org/jira/browse/YARN-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14568606#comment-14568606 ] Hadoop QA commented on YARN-3755: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | pre-patch | 15m 43s | Pre-patch trunk compilation is healthy. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | | {color:green}+1{color} | javac | 7m 35s | There were no new javac warning messages. | | {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:red}-1{color} | checkstyle | 0m 37s | The applied patch generated 3 new checkstyle issues (total was 58, now 60). | | {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that end in whitespace. | | {color:green}+1{color} | install | 1m 34s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:green}+1{color} | findbugs | 1m 13s | The patch does not introduce any new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | yarn tests | 6m 9s | Tests passed in hadoop-yarn-server-nodemanager. | | | | 43m 32s | | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12736742/YARN-3755-1.patch | | Optional Tests | javadoc javac unit findbugs checkstyle | | git revision | trunk / 990078b | | checkstyle | https://builds.apache.org/job/PreCommit-YARN-Build/8160/artifact/patchprocess/diffcheckstylehadoop-yarn-server-nodemanager.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8160/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8160/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8160/console | This message was automatically generated. > Log the command of launching containers > --- > > Key: YARN-3755 > URL: https://issues.apache.org/jira/browse/YARN-3755 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 2.7.0 >Reporter: Jeff Zhang >Assignee: Jeff Zhang > Attachments: YARN-3755-1.patch > > > In the resource manager log, yarn would log the command for launching AM, > this is very useful. But there's no such log in the NN log for launching > containers. It would be difficult to diagnose when containers fails to launch > due to some issue in the commands. Although user can look at the commands in > the container launch script file, this is an internal things of yarn, usually > user don't know that. In user's perspective, they only know what commands > they specify when building yarn application. > {code} > 2015-06-01 16:06:42,245 INFO > org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Command > to launch container container_1433145984561_0001_01_01 : > $JAVA_HOME/bin/java -server -Djava.net.preferIPv4Stack=true > -Dhadoop.metrics.log.level=WARN -Xmx1024m > -Dlog4j.configuratorClass=org.apache.tez.common.TezLog4jConfigurator > -Dlog4j.configuration=tez-container-log4j.properties > -Dyarn.app.container.log.dir= -Dtez.root.logger=info,CLA > -Dsun.nio.ch.bugLevel='' org.apache.tez.dag.app.DAGAppMaster > 1>/stdout 2>/stderr > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)