[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14523987#comment-14523987
]
Xuan Gong commented on YARN-1906:
-
Close this as Can not re-produce. Feel free to re-open it if you see the
testcase failure again.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Test
Components: test
Affects Versions: 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989883#comment-13989883
]
Ashwin Shankar commented on YARN-1906:
--
Hey [~mitdesai], I encountered this issue in my pre-commit build,but it seemed
to have
happened at a different place in this test. Here is the link:
https://builds.apache.org/job/PreCommit-YARN-Build/3686//testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMRestart/testQueueMetricsOnRMRestart/
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13989915#comment-13989915
]
Mit Desai commented on YARN-1906:
-
[~zjshen] and [~wangda], Totally agree. We should add the message for the
assertion in with the fix.
[~ashwinshankar77]: Thanks for pointing this out. I also got the failure on
this line last time. Additionally, I ran the test couple of time and found out
that the test was failing randomly on any of the asserts. Still trying to
figure out where the race is.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13988356#comment-13988356
]
Wangda Tan commented on YARN-1906:
--
I just encountered this issue when I working on YARN-1201 too,
The assertion failure code snippet,
{code}
while (loadedApp1.getAppAttempts().size() != 2) {
Thread.sleep(200);
}
attempt1 = loadedApp1.getCurrentAppAttempt();
attemptId1 = attempt1.getAppAttemptId();
rm2.waitForState(attemptId1, RMAppAttemptState.SCHEDULED);
assertQueueMetrics(qm2, 1, 1, 0, 0);
{code}
And in assertQueueMetrics(), following assertion is failed
{code}
Assert.assertEquals(qm.getAppsSubmitted(),
appsSubmitted + appsSubmittedCarryOn);
{code}
+1 to [~zjshen]'s suggestion, we should add message to assertion sentence
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Test
Affects Versions: 2.4.0
Reporter: Mit Desai
Assignee: Mit Desai
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965449#comment-13965449
]
Mit Desai commented on YARN-1906:
-
*Explanation of the changes made*
# Two assert statements were removed from the test that were verifying the
pending application count increased from 0 to 1. This is an intermediate result
(which is good to test if consistent). In this case, the intermediate results
are inconsistent as the application transition to pending state can be or
cannot be detected when the assert is called. The aim of the test is to check
the queueMetrics value before and after restart. And this is working as
expected without the assert for pendingApps. I have tested the patch by running
the test 25 times and it passes.
# The assertQueueMetrics function was not properly implemented. assertEquals()
takes in 2 parameters 1. expected value, 2. Actual value. All the asserts in
the assertQueueMetrics were implemented in the opposite way leading to a wrong
error message on an assert failure.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965492#comment-13965492
]
Zhijie Shen commented on YARN-1906:
---
1. I looked into exception again:
{code}
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{code}
It seems the test fails at
{code}
// finish the AMs
finishApplicationMaster(loadedApp1, rm2, nm1, am1);
assertQueueMetrics(qm2, 1, 0, 0, 1);
{code}
Race condition here? Should we waitForState here before assertion?
2. One suggestion on assertQueueMetrics: It would be better to add the messages
for the for assertion sentences, such that when an exception happens, we can
easily see which metric is wrong.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13965500#comment-13965500
]
Hadoop QA commented on YARN-1906:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12639589/YARN-1906.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/3542//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3542//console
This message is automatically generated.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch, YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963114#comment-13963114
]
Mit Desai commented on YARN-1906:
-
Thanks for the feedback Jon. I will take a look today
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963491#comment-13963491
]
Zhijie Shen commented on YARN-1906:
---
Maybe we can wait until, for example, the appSubmitted, is changed (increased
or decreased by 1)?
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13963539#comment-13963539
]
Mit Desai commented on YARN-1906:
-
I was going through the app state transitions and I have the following
conclusions:
* The state form NEW to SUBMITTED and SUBMITTED to ACCEPTED is almost
instantaneous.
* When assertQueueMetrics is called, the app state I found was always ACCEPTED.
The next state that it can transition into is RUNNING /KILLING/FINAL_SAVING.
Which will not be changed until the scheduler picks up the app.
[~jeagles], So we will not be able to use the waitForState method here.
[~zjshen], When the app is submitted and we check the numbers in the
assertQueueMetrics method, the appSubmitted will already have been 1, so
waiting for it to increment will be an infinite waiting. Moreover, in the
assertQueueMetrics we verifying the same thing (i.e the number of submitted
apps and the number of apps in pending state is what we expect it to be).
I do not have another solution for the problem. May be we need to think some
other ways.
Please provide your feedback if you have other ideas or think I am heading in
some wrong direction.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-1906) TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and branch2
[
https://issues.apache.org/jira/browse/YARN-1906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13962175#comment-13962175
]
Jonathan Eagles commented on YARN-1906:
---
Mit, you might consider using waitForState instead of a raw sleep. This will
protected us in the case of a missed race condition, though perhaps will result
in more sleep time overall.
TestRMRestart#testQueueMetricsOnRMRestart fails intermittently on trunk and
branch2
---
Key: YARN-1906
URL: https://issues.apache.org/jira/browse/YARN-1906
Project: Hadoop YARN
Issue Type: Bug
Reporter: Mit Desai
Assignee: Mit Desai
Fix For: 3.0.0, 2.5.0
Attachments: YARN-1906.patch
Here is the output of the format
{noformat}
testQueueMetricsOnRMRestart(org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart)
Time elapsed: 9.757 sec FAILURE!
java.lang.AssertionError: expected:2 but was:1
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.assertQueueMetrics(TestRMRestart.java:1735)
at
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart.testQueueMetricsOnRMRestart(TestRMRestart.java:1706)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
