[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171828#comment-15171828
]
Takashi Ohnishi commented on YARN-4566:
---
Thank you [~rohithsharma] for reviewing and committing !!
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Fix For: 2.9.0
>
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171332#comment-15171332
]
Hudson commented on YARN-4566:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #9388 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/9388/])
YARN-4566. Fix test failure in TestMiniYarnClusterNodeUtilization.
(rohithsharmaks: rev e0b14f26f5201a149218276469434df511697acc)
*
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-tests/src/test/java/org/apache/hadoop/yarn/server/TestMiniYarnClusterNodeUtilization.java
* hadoop-yarn-project/CHANGES.txt
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15171278#comment-15171278
]
Rohith Sharma K S commented on YARN-4566:
-
+1LGTM
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15090664#comment-15090664
]
Takashi Ohnishi commented on YARN-4566:
---
The test failed with another cause...
The stack trace is the same with YARN-4453
{code}
java.lang.Exception: test timed out after 6 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:158)
at com.sun.proxy.$Proxy85.nodeHeartbeat(Unknown Source)
at
org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:113)
{code}
And also the TestContainerManagerSecurity failed with the same error reported
by YARN-4342
{code}
java.lang.Exception: test timed out after 12 milliseconds
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:158)
at com.sun.proxy.$Proxy88.startContainers(Unknown Source)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.startContainer(TestContainerManagerSecurity.java:556)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testStartContainer(TestContainerManagerSecurity.java:477)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:249)
at
org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:157)
{code}
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: test
>Reporter: Takashi Ohnishi
>Assignee: Takashi Ohnishi
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089396#comment-15089396
]
Hadoop QA commented on YARN-4566:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
8s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 5s {color} |
{color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 6s {color} |
{color:red} hadoop-yarn-server-tests in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
| JDK v1.7.0_91 Failed junit tests |
hadoop.yarn.server.TestContainerManagerSecurity |
| | hadoop.yarn.server.TestMiniYarnClusterNodeUtilization |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12781227/YARN-4566.1.patch |
| JIRA Issue | YARN-4566 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 049ee168d65b
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089334#comment-15089334
]
Takashi Ohnishi commented on YARN-4566:
---
By the way, is it better to change getNodeUtilization not to return null?
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Takashi Ohnishi
> Attachments: YARN-4566.1.patch
>
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-4566) TestMiniYarnClusterNodeUtilization sometimes fails on trunk
[
https://issues.apache.org/jira/browse/YARN-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15089328#comment-15089328
]
Takashi Ohnishi commented on YARN-4566:
---
The NPE happens at getNodeUtilization:
{code}
213 // Give the heartbeat time to propagate to the RM (max 10 seconds)
214 // We check if the nodeUtilization is up to date
215 for (int i=0; i<100; i++) {
216 for (RMNode ni : rmContext.getRMNodes().values()) {
217 if (ni.getNodeUtilization().equals(nodeUtilization)) {
218 break;
219 }
220 }
221 Thread.sleep(100);
222 }
{code}
Looking the code of NodeResourceMonitorImpl.java, the instance returned by
getNodeUtilization keeps changing periodically.
I'm not sure but this seems to be the cause of NPE.
In my environment, this failure disappear by adding a null check of the return
value.
I will attache a patch.
> TestMiniYarnClusterNodeUtilization sometimes fails on trunk
> ---
>
> Key: YARN-4566
> URL: https://issues.apache.org/jira/browse/YARN-4566
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Reporter: Takashi Ohnishi
>
> TestMiniYarnClusterNodeUtilization often fails with NPE.
> {code}
> testUpdateNodeUtilization(org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization)
> Time elapsed: 3.752 sec <<< ERROR!
> java.lang.NullPointerException: null
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.verifySimulatedUtilization(TestMiniYarnClusterNodeUtilization.java:217)
> at
> org.apache.hadoop.yarn.server.TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization(TestMiniYarnClusterNodeUtilization.java:116)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
