[
https://issues.apache.org/jira/browse/MAPREDUCE-2905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150964#comment-13150964
]
Jeff Bean commented on MAPREDUCE-2905:
--------------------------------------
testFairScheduler is buggy. In one instance it actually looks for the bug
condition and passes it by asserting that tt1 fills to capacity on small jobs
that shouldn't clump together:
checkAssignment("tt1", "attempt_test_0001_m_000000_0 on tt1",
"attempt_test_0001_r_000000_0 on tt1",
"attempt_test_0002_m_000000_0 on tt1",
"attempt_test_0002_r_000000_0 on tt1");
checkAssignment("tt2", "attempt_test_0001_m_000001_0 on tt2",
"attempt_test_0002_r_000001_0 on tt2");
If tt1 has a capacity of 2 mappers and 2 reducers, I expect 1 mapper and 1
reducer to be assigned each as to assign all 4 slots would be the clumping
condition that this patch is trying to address.
On every instance of checkAssignment that gets multiple tasks, the test fails
on the length of the task list. The assignMultiple tests are broken for the
reason I describe above. The other tests that fail are the
testDelaySchedulingAtNodeLevel, testDelaySchedulingAtRackLevel, and
testDelaySchedulingOffRack. In all three of those cases a task list length of 2
of the same type of is expected and a task list length of 1 is returned. Again,
with a default capacity of 2 slots per task type, we'd be over capacity and
actually looking for the condition this patch fixes.
Relevant output from failure:
Testcase: testSmallJobsWithAssignMultiple took 0.58 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testSmallJobsWithAssignMultiple(TestFairScheduler.java:784)
Testcase: testLargeJobs took 0.432 sec
Testcase: testLargeJobsWithAssignMultiple took 0.435 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithAssignMultiple(TestFairScheduler.java:954)
Testcase: testJobsWithPriorities took 0.432 sec
Testcase: testLargeJobsWithPools took 0.672 sec
Testcase: testLargeJobsWithExcessCapacity took 0.598 sec
Testcase: testLargeJobsWithExcessCapacityAndAssignMultiple took 0.594 sec
FAILED
expected:<4> but was:<2>
junit.framework.AssertionFailedError: expected:<4> but was:<2>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testLargeJobsWithExcessCapacityAndAssignMultiple(TestFairScheduler.java:1301)
Testcase: testSmallJobInLargePool took 0.517 sec
Testcase: testPoolMaxJobs took 0.842 sec
Testcase: testUserMaxJobs took 0.73 sec
Testcase: testComplexJobLimits took 2.053 sec
Testcase: testSizeBasedWeight took 0.387 sec
Testcase: testPoolWeights took 0.851 sec
Testcase: testPoolWeightsWhenNoMaps took 0.637 sec
Testcase: testPoolMaxMapsReduces took 0.404 sec
Testcase: testCapBasedLoadManager took 0.119 sec
Testcase: testMinSharePreemption took 0.487 sec
Testcase: testMinSharePreemptionWithSmallJob took 0.426 sec
Testcase: testFairSharePreemption took 0.612 sec
Testcase: testFairSharePreemptionFromMultiplePools took 0.602 sec
Testcase: testMinAndFairSharePreemption took 0.42 sec
Testcase: testNoPreemptionIfDisabled took 0.457 sec
Testcase: testNoPreemptionIfOnlyLogging took 0.458 sec
Testcase: testDelaySchedulingAtNodeLevel took 0.366 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtNodeLevel(TestFairScheduler.java:2301)
Testcase: testDelaySchedulingAtRackLevel took 0.521 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
at
org.apache.hadoop.mapred.TestFairScheduler.checkAssignment(TestFairScheduler.java:2810)
at
org.apache.hadoop.mapred.TestFairScheduler.testDelaySchedulingAtRackLevel(TestFairScheduler.java:2349)
Testcase: testDelaySchedulingOffRack took 0.261 sec
FAILED
expected:<2> but was:<1>
junit.framework.AssertionFailedError: expected:<2> but was:<1>
This is every time checkAssignment() gets called with more than one task as a
parameter. This is a buggy test. For example, in one instance it actuall
> CapBasedLoadManager incorrectly allows assignment when assignMultiple is true
> (was: assignmultiple per job)
> -----------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2905
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2905
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: contrib/fair-share
> Affects Versions: 0.20.2
> Reporter: Jeff Bean
> Assignee: Jeff Bean
> Attachments: MR-2905.10-13-2011, MR-2905.patch, MR-2905.patch.2,
> mr-2905.txt, screenshot-1.jpg
>
>
> We encountered a situation where in the same cluster, large jobs benefit from
> mapred.fairscheduler.assignmultiple, but small jobs with small numbers of
> mappers do not: the mappers all clump to fully occupy just a few nodes, which
> causes those nodes to saturate and bottleneck. The desired behavior is to
> spread the job across more nodes so that a relatively small job doesn't
> saturate any node in the cluster.
> Testing has shown that setting mapred.fairscheduler.assignmultiple to false
> gives the desired behavior for small jobs, but is unnecessary for large jobs.
> However, since this is a cluster-wide setting, we can't properly tune.
> It'd be nice if jobs can set a param similar to
> mapred.fairscheduler.assignmultiple on submission to better control the task
> distribution of a particular job.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
