[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124168#comment-16124168
]
Hudson commented on YARN-6967:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12170 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/12170/])
YARN-6967. Limit application attempt's diagnostic message size (templedf: rev
65364defb4a633ca20b39ebc38cd9c0db63a5835)
* (edit)
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: YARN-6967.01.patch
>
>
> YARN-6125 implemented {{BoundedAppender}} and applied to the field
> {{diagnostics}} to limit the diagnostic message's size.
> However, some code bypasses this limit. In
> {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local
> variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore
> is used.
> A simple fix is to also use {{BoundedAppender}} for the local variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124108#comment-16124108
]
Daniel Templeton commented on YARN-6967:
OK, LGTM +1
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Attachments: YARN-6967.01.patch
>
>
> YARN-6125 implemented {{BoundedAppender}} and applied to the field
> {{diagnostics}} to limit the diagnostic message's size.
> However, some code bypasses this limit. In
> {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local
> variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore
> is used.
> A simple fix is to also use {{BoundedAppender}} for the local variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16122652#comment-16122652
]
Chengbing Liu commented on YARN-6967:
-
[~templedf] [~andras.piros] Could you review the patch again? Thanks!
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Attachments: YARN-6967.01.patch
>
>
> YARN-6125 implemented {{BoundedAppender}} and applied to the field
> {{diagnostics}} to limit the diagnostic message's size.
> However, some code bypasses this limit. In
> {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local
> variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore
> is used.
> A simple fix is to also use {{BoundedAppender}} for the local variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16119291#comment-16119291
]
Chengbing Liu commented on YARN-6967:
-
Hi [~templedf], I don't think the message will be any longer. In the patch, the
local {{BoundedAppender diags}} is independent from the field variable
{{diagnostics}} of {{RMAppAttemptImpl}}, and starts as a new
{{BoundedAppender}}.
We have seen cases where a Spark application throws a large exception message,
which is too large for ZooKeeper. Then both the AppMaster and the
ResourceManager keep trying and failing, making the RM irresponsive in the end.
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Attachments: YARN-6967.01.patch
>
>
> YARN-6125 implemented {{BoundedAppender}} and applied to the field
> {{diagnostics}} to limit the diagnostic message's size.
> However, some code bypasses this limit. In
> {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local
> variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore
> is used.
> A simple fix is to also use {{BoundedAppender}} for the local variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118522#comment-16118522
]
Daniel Templeton commented on YARN-6967:
Thanks for catching that, [~chengbing.liu]. In the original code the diags
string is only set, never appended to. In the patch, you're only appending.
Seems like that might change the diag messages and tend to make them longer on
average (though never longer than the limit).
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Chengbing Liu
> Attachments: YARN-6967.01.patch
>
>
> YARN-6125 implemented {{BoundedAppender}} and applied to the field
> {{diagnostics}} to limit the diagnostic message's size.
> However, some code bypasses this limit. In
> {{RMAppAttemptImpl.rememberTargetTransitionsAndStoreState(...)}}, a local
> variable {{diags}} will finally be written into ZooKeeper if ZKRMStateStore
> is used.
> A simple fix is to also use {{BoundedAppender}} for the local variable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6967) Limit application attempt's diagnostic message size thoroughly
[
https://issues.apache.org/jira/browse/YARN-6967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118323#comment-16118323
]
Hadoop QA commented on YARN-6967:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 44m
28s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 29s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Issue | YARN-6967 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12880817/YARN-6967.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 864f1b87af67 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31
14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 9891295 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/16770/testReport/ |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/16770/console |
| Powered by | Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Limit application attempt's diagnostic message size thoroughly
> --
>
> Key: YARN-6967
> URL: https://issues.apache.org/jira/browse/YARN-6967
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 2.8.1
>Reporter: Chengbing Liu
>Assignee: Ch
