zhihai xu commented on YARN-2666:

Hi [~ywskycn], Could you assign this JIRA to me?
I think I know what cause this Intermittent failure.
The problem is because ContinuousSchedulingThread is calling 
continuousSchedulingAttempt periodically. 
And continuousSchedulingAttempt doesn't hold the FairScheduler lock.
continuousSchedulingAttempt can run at any time,
    for (NodeId nodeId : nodeIdList) {
      FSSchedulerNode node = getFSSchedulerNode(nodeId);
      try {
        if (node != null && Resources.fitsIn(minimumAllocation,
            node.getAvailableResource())) {
      } catch (Throwable ex) {
        LOG.error("Error while attempting scheduling for node " + node +
            ": " + ex.toString(), ex);
when the testContinuousScheduling run scheduler.allocate to make a container 
allocation request.
It is possible application.updateResourceRequests in scheduler.allocate is 
running right after attemptScheduling first node and before attemptScheduling 
second node. then the second node with less resource will allocate container 
for this allocation request.
Then the issue will happen: both containers are allocated on the same node.
The default ContinuousSchedulingSleepMs is 5ms which is very short, If we 
increase ContinuousSchedulingSleepMs, the test failure will be much less. We 
can make the test deterministic by manually calling continuousSchedulingAttempt 
after second allocation request and stopping the ContinuousSchedulingThread 
before second allocation request.
I uploaded a patch which will stop ContinuousSchedulingThread before second 
allocation request and manually call continuousSchedulingAttempt after second 
allocation request.

> TestFairScheduler.testContinuousScheduling fails Intermittently
> ---------------------------------------------------------------
>                 Key: YARN-2666
>                 URL: https://issues.apache.org/jira/browse/YARN-2666
>             Project: Hadoop YARN
>          Issue Type: Test
>          Components: scheduler
>            Reporter: Tsuyoshi Ozawa
>            Assignee: Wei Yan
>         Attachments: YARN-2666.000.patch
> The test fails on trunk.
> {code}
> Tests run: 79, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.698 sec 
> <<< FAILURE! - in 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
> testContinuousScheduling(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
>   Time elapsed: 0.582 sec  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
>       at org.junit.Assert.fail(Assert.java:88)
>       at org.junit.Assert.failNotEquals(Assert.java:743)
>       at org.junit.Assert.assertEquals(Assert.java:118)
>       at org.junit.Assert.assertEquals(Assert.java:555)
>       at org.junit.Assert.assertEquals(Assert.java:542)
>       at 
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testContinuousScheduling(TestFairScheduler.java:3372)
> {code}

This message was sent by Atlassian JIRA

Reply via email to