[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144424#comment-14144424 ] Sandy Ryza commented on YARN-2252: -- +1 > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch, yarn-2252-2.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142745#comment-14142745 ] Wei Yan commented on YARN-2252: --- Patch looks good to me. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch, yarn-2252-2.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142720#comment-14142720 ] Karthik Kambatla commented on YARN-2252: The patch touches only TestFairScheduler, and couldn't have caused all these test failures. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch, yarn-2252-2.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[
https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142718#comment-14142718
]
Hadoop QA commented on YARN-2252:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670318/yarn-2252-2.patch
against trunk revision c50fc92.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The following test timeouts occurred in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
org.apache.hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart
org.apache.hadoop.yarn.server.resourcemanager.security.TestRMDelegationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart
org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
org.apache.hadoop.yarn.server.resourcemanager.TestKillApplicationWithRMHA
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestContainerResourceUsage
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/5066//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5066//console
This message is automatically generated.
> Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
>
>
> Key: YARN-2252
> URL: https://issues.apache.org/jira/browse/YARN-2252
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: trunk-win
>Reporter: Ratandeep Ratti
> Labels: hadoop2, scheduler, yarn
> Attachments: YARN-2252-1.patch, yarn-2252-2.patch
>
>
> This test-case is failing sporadically on my machine. I think I have a
> plausible explanation for this.
> It seems that when the Scheduler is being asked for resources, the resource
> requests that are being constructed have no preference for the hosts (nodes).
> The two mock hosts constructed, both have a memory of 8192 mb.
> The containers(resources) being requested each require a memory of 1024mb,
> hence a single node can execute both the resource requests for the
> application.
> In the end of the test-case it is being asserted that the containers
> (resource requests) be executed on different nodes, but since we haven't
> specified any preferences for nodes when requesting the resources, the
> scheduler (at times) executes both the containers (requests) on the same node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142198#comment-14142198 ] Wei Yan commented on YARN-2252: --- +1 for the proposal, [~kasha]. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[
https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141515#comment-14141515
]
Hadoop QA commented on YARN-2252:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654298/YARN-2252-1.patch
against trunk revision 9e35571.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5055//console
This message is automatically generated.
> Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
>
>
> Key: YARN-2252
> URL: https://issues.apache.org/jira/browse/YARN-2252
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: trunk-win
>Reporter: Ratandeep Ratti
> Labels: hadoop2, scheduler, yarn
> Attachments: YARN-2252-1.patch
>
>
> This test-case is failing sporadically on my machine. I think I have a
> plausible explanation for this.
> It seems that when the Scheduler is being asked for resources, the resource
> requests that are being constructed have no preference for the hosts (nodes).
> The two mock hosts constructed, both have a memory of 8192 mb.
> The containers(resources) being requested each require a memory of 1024mb,
> hence a single node can execute both the resource requests for the
> application.
> In the end of the test-case it is being asserted that the containers
> (resource requests) be executed on different nodes, but since we haven't
> specified any preferences for nodes when requesting the resources, the
> scheduler (at times) executes both the containers (requests) on the same node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[
https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14141512#comment-14141512
]
Karthik Kambatla commented on YARN-2252:
The fix is specific to testContinuousScheduling. I propose fixing all the tests
under TestFairScheduler, by using the same global field {{scheduler}} in all
the tests even when the test method initializes a new instance of the
scheduler. The teardown method should also stop {{resourceManager}} if not
null.
We should follow up this work with (1) re-using the scheduler instance within
{{resourceManager}} and (2) moving out tests with different FS configurations
to different files.
> Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
>
>
> Key: YARN-2252
> URL: https://issues.apache.org/jira/browse/YARN-2252
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: trunk-win
>Reporter: Ratandeep Ratti
> Labels: hadoop2, scheduler, yarn
> Attachments: YARN-2252-1.patch
>
>
> This test-case is failing sporadically on my machine. I think I have a
> plausible explanation for this.
> It seems that when the Scheduler is being asked for resources, the resource
> requests that are being constructed have no preference for the hosts (nodes).
> The two mock hosts constructed, both have a memory of 8192 mb.
> The containers(resources) being requested each require a memory of 1024mb,
> hence a single node can execute both the resource requests for the
> application.
> In the end of the test-case it is being asserted that the containers
> (resource requests) be executed on different nodes, but since we haven't
> specified any preferences for nodes when requesting the resources, the
> scheduler (at times) executes both the containers (requests) on the same node.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14064467#comment-14064467 ] Ratandeep Ratti commented on YARN-2252: --- Thanks Wei! > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14063846#comment-14063846 ] Wei Yan commented on YARN-2252: --- [~rdsr], sorry for later reply. the patch looks good to me. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[
https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14054803#comment-14054803
]
Hadoop QA commented on YARN-2252:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12654298/YARN-2252-1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-YARN-Build/4217//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4217//console
This message is automatically generated.
> Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
>
>
> Key: YARN-2252
> URL: https://issues.apache.org/jira/browse/YARN-2252
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: trunk-win
>Reporter: Ratandeep Ratti
> Labels: hadoop2, scheduler, yarn
> Attachments: YARN-2252-1.patch
>
>
> This test-case is failing sporadically on my machine. I think I have a
> plausible explanation for this.
> It seems that when the Scheduler is being asked for resources, the resource
> requests that are being constructed have no preference for the hosts (nodes).
> The two mock hosts constructed, both have a memory of 8192 mb.
> The containers(resources) being requested each require a memory of 1024mb,
> hence a single node can execute both the resource requests for the
> application.
> In the end of the test-case it is being asserted that the containers
> (resource requests) be executed on different nodes, but since we haven't
> specified any preferences for nodes when requesting the resources, the
> scheduler (at times) executes both the containers (requests) on the same node.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053790#comment-14053790 ] Wei Yan commented on YARN-2252: --- Thanks, [~rdsr]. I'll take a look later. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > Attachments: YARN-2252-1.patch > > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053413#comment-14053413 ] Ratandeep Ratti commented on YARN-2252: --- Wei, calling fairscheduler.stop() will not stop the threads. It seems that the threads "continuousScheduling thread" and "update thread" are not handling interrupt properly. Though we are doing [schedulingthread | updateThread].interrupt(), we do need to keep checking the interrupt flag in the while loop of these threads. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti >Assignee: Wei Yan > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053280#comment-14053280 ] Ratandeep Ratti commented on YARN-2252: --- Wei, while analyzing this failure, I stumbled upon another potential problem. The method SchedulerApplication.getCurrentConsumption() is not synchronized, this may lead to visibility (staleness) problems > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti >Assignee: Wei Yan > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053168#comment-14053168 ] Wei Yan commented on YARN-2252: --- Look the code again. I remembered why I created a new FS, as the old one doesn't have continuous scheduling enabled. Yes, the problem is the testLoadConfigurationOnInitialize(). We need to do the fairscheduler.stop() before we shut down the test. Otherwise, the update thread and continuousScheduling thread are still running. I'll update with a patch later. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053157#comment-14053157 ] Wei Yan commented on YARN-2252: --- Yes, that makes sense. We can only use the old scheduler, instead of creating a new one inside the testContinuousScheduling. Cannot remember why I created a new fairscheduler why I wrote the test case. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053129#comment-14053129 ] Ratandeep Ratti commented on YARN-2252: --- Thanks for the help Wei. I think I might have found the reason for this failure. Notice that the test-case TestFairScheduler.testLoadConfigurationOnInitialize() first starts up the continuousScheduling thread (under FairScheduler object). When the execution flow reaches the test-case TestFairScheduler.testContinuousScheduling, even though we create a new FairScheduler object, the old thread (continuousScheduling) is still live, and now, there are now two live continuousScheduling threads trying to assign containers to nodes. The problem does not seem to occur when I properly stop each thread before starting another execution of this test-case. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[ https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053039#comment-14053039 ] Wei Yan commented on YARN-2252: --- Thanks for the catch, [~rdsr]. For the node perference, scheduler always tries to assign new container to the node with the most available resource, YARN-1290. I also replayed the test failure locally. I think it may due to the node.availableResource's update. I'll check out it. > Intermittent failure for testcase TestFairScheduler.testContinuousScheduling > > > Key: YARN-2252 > URL: https://issues.apache.org/jira/browse/YARN-2252 > Project: Hadoop YARN > Issue Type: Bug > Components: scheduler >Affects Versions: trunk-win >Reporter: Ratandeep Ratti > Labels: hadoop2, scheduler, yarn > > This test-case is failing sporadically on my machine. I think I have a > plausible explanation for this. > It seems that when the Scheduler is being asked for resources, the resource > requests that are being constructed have no preference for the hosts (nodes). > The two mock hosts constructed, both have a memory of 8192 mb. > The containers(resources) being requested each require a memory of 1024mb, > hence a single node can execute both the resource requests for the > application. > In the end of the test-case it is being asserted that the containers > (resource requests) be executed on different nodes, but since we haven't > specified any preferences for nodes when requesting the resources, the > scheduler (at times) executes both the containers (requests) on the same node. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (YARN-2252) Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
[
https://issues.apache.org/jira/browse/YARN-2252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14052398#comment-14052398
]
Ratandeep Ratti commented on YARN-2252:
---
Pasting test-case failure message
{quote}
Running
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 180.775 sec <<<
FAILURE! - in org.apache.hadoop.yarn.server.resour
tFairScheduler
testContinuousScheduling(org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler)
Time elapsed: 180.495 se
java.lang.AssertionError: expected:<2> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.TestFairScheduler.testContinuousScheduling(TestFairSchedul
Results :
Failed tests:
TestFairScheduler.testContinuousScheduling:2758 expected:<2> but was:<1>
{quote}
> Intermittent failure for testcase TestFairScheduler.testContinuousScheduling
>
>
> Key: YARN-2252
> URL: https://issues.apache.org/jira/browse/YARN-2252
> Project: Hadoop YARN
> Issue Type: Bug
> Components: scheduler
>Affects Versions: trunk-win
>Reporter: Ratandeep Ratti
> Labels: hadoop2, scheduler, yarn
>
> This test-case is failing sporadically on my machine. I think I have a
> plausible explanation for this.
> It seems that when the Scheduler is being asked for resources, the resource
> requests that are being constructed have no preference for the hosts (nodes).
> The two mock hosts constructed, both have a memory of 8192 mb.
> The containers(resources) being requested each require a memory of 1024mb,
> hence a single node can execute both the resource requests for the
> application.
> In the end of the test-case it is being asserted that the containers
> (resource requests) be executed on different nodes, but since we haven't
> specified any preferences for nodes when requesting the resources, the
> scheduler (at times) executes both the containers (requests) on the same node.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
