[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[ https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127873#comment-15127873 ] Sidharta Seethana commented on YARN-4649: - Thanks for the review and commit, [~vvasudev]. > Add additional logging to some NM state store operations > > > Key: YARN-4649 > URL: https://issues.apache.org/jira/browse/YARN-4649 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.9.0 > > Attachments: YARN-4649.001.patch, YARN-4649.002.patch > > > Adding additional logging to NM container recovery code (specifically > application/container status operations) makes it easier to debug container > recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[ https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15127631#comment-15127631 ] Hudson commented on YARN-4649: -- FAILURE: Integrated in Hadoop-trunk-Commit #9221 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9221/]) YARN-4649. Add additional logging to some NM state store operations. (vvasudev: rev 1cd55e0c171f7c4dec6f843931285557d59cd5ea) * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/ContainerManagerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMLeveldbStateStoreService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/recovery/NMStateStoreService.java > Add additional logging to some NM state store operations > > > Key: YARN-4649 > URL: https://issues.apache.org/jira/browse/YARN-4649 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Fix For: 2.9.0 > > Attachments: YARN-4649.001.patch, YARN-4649.002.patch > > > Adding additional logging to NM container recovery code (specifically > application/container status operations) makes it easier to debug container > recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[ https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15125256#comment-15125256 ] Varun Vasudev commented on YARN-4649: - The single log line thing was to make grep and log parsing easier. +1 for the latest patch. I'll commit this tomorrow if no one objects. > Add additional logging to some NM state store operations > > > Key: YARN-4649 > URL: https://issues.apache.org/jira/browse/YARN-4649 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Sidharta Seethana >Assignee: Sidharta Seethana >Priority: Minor > Attachments: YARN-4649.001.patch, YARN-4649.002.patch > > > Adding additional logging to NM container recovery code (specifically > application/container status operations) makes it easier to debug container > recovery related issues. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[
https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15123332#comment-15123332
]
Varun Vasudev commented on YARN-4649:
-
Thanks for the patch [~sidharta-s]. Some feedack -
1) Instead of
{code}
+if (LOG.isDebugEnabled()) {
+ LOG.debug("Recovering container with state: ");
+ LOG.debug("Diagnostics: " + rcs.getDiagnostics());
+ LOG.debug("Exit Code: " + rcs.getExitCode());
+ LOG.debug("Start Request: " + rcs.getStartRequest());
+ LOG.debug("Status: " + rcs.getStatus());
+}
{code}
can you add a toString() method to RecoveredContainerState and use that when
logging? That way if someone adds a new field, it'll get reflected in the logs
automatically
2) For the following statements, can you merge them to one log line?
{code}
+if (LOG.isDebugEnabled()) {
+ LOG.debug("storeContainer.containerId: " + containerId );
+ LOG.debug("storeContainer.startRequest: " + startRequest);
+}
{code}
and
{code}
+if (LOG.isDebugEnabled()) {
+ LOG.debug("storeContainerDiagnostics.containerId: " + containerId);
+ LOG.debug("storeContainerDiagnostics.diagnostics: " + diagnostics);
+}
{code}
and
{code}
+if (LOG.isDebugEnabled()) {
+ LOG.debug("storeContainerResourceChanged.containerId: " + containerId);
+ LOG.debug("storeContainerResourceChanged.capability: " + capability);
+}
{code}
and
{code}
+if (LOG.isDebugEnabled()) {
+ LOG.debug("storeApplication.appId: " + appId);
+ LOG.debug("storeApplication.proto: " + p);
+}
{code}
> Add additional logging to some NM state store operations
>
>
> Key: YARN-4649
> URL: https://issues.apache.org/jira/browse/YARN-4649
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-4649.001.patch
>
>
> Adding additional logging to NM container recovery code (specifically
> application/container status operations) makes it easier to debug container
> recovery related issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[
https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124438#comment-15124438
]
Hadoop QA commented on YARN-4649:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
1s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 19s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 23s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 15s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
patch generated 4 new + 130 unchanged - 0 fixed = 134 total (was 130) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 38s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 17s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 33m 32s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12785273/YARN-4649.002.patch |
| JIRA Issue | YARN-4649 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux fed7bac621a6 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[
https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15124057#comment-15124057
]
Sidharta Seethana commented on YARN-4649:
-
Thanks for the review, [~vvasudev]. I'll add a {{toString()}} function to
{{RecoveredContainerState}} . Regarding 2), could you please clarify why the
log statements need to be combined? They are structured this way for better
readability. In addition, the second log line in each case could result in
multiple log lines anyway.
> Add additional logging to some NM state store operations
>
>
> Key: YARN-4649
> URL: https://issues.apache.org/jira/browse/YARN-4649
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Sidharta Seethana
>Assignee: Sidharta Seethana
>Priority: Minor
> Attachments: YARN-4649.001.patch
>
>
> Adding additional logging to NM container recovery code (specifically
> application/container status operations) makes it easier to debug container
> recovery related issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4649) Add additional logging to some NM state store operations
[
https://issues.apache.org/jira/browse/YARN-4649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15120977#comment-15120977
]
Hadoop QA commented on YARN-4649:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
30s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 28s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 21s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
23s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 20s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 24s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 13s
{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:
patch generated 1 new + 61 unchanged - 0 fixed = 62 total (was 61) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 20s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 52s {color}
| {color:red} hadoop-yarn-server-nodemanager in the patch failed with JDK
v1.7.0_91. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 31m 48s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12784798/YARN-4649.001.patch |
| JIRA Issue | YARN-4649 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux f9eaceffdb44 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
