[
https://issues.apache.org/jira/browse/MAPREDUCE-936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12751394#action_12751394
]
Hadoop QA commented on MAPREDUCE-936:
-------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12418583/MAPREDUCE-936.2.patch
against trunk revision 811134.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/38/testReport/
Findbugs warnings:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/38/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/38/artifact/trunk/build/test/checkstyle-errors.html
Console output:
http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h6.grid.sp2.yahoo.net/38/console
This message is automatically generated.
> Allow a load difference in fairshare scheduler
> ----------------------------------------------
>
> Key: MAPREDUCE-936
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-936
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 0.20.1, 0.21.0, 0.22.0
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Attachments: MAPREDUCE-936.1.patch, MAPREDUCE-936.2.patch
>
>
> The problem we are facing: It takes a long time for all tasks of a job to get
> scheduled on the cluster, even if the cluster is almost empty.
> There are two reasons that together lead to this situation:
> 1. The load factor makes sure each TT runs the same number of tasks. (This is
> the part that this patch tries to change).
> 2. The scheduler tries to schedule map tasks locally (first node-local, then
> rack-local). There is a wait time (mapred.fairscheduler.localitywait.node and
> mapred.fairscheduler.localitywait.rack, both are around 10 sec in our conf),
> and accumulated wait time (JobInfo.localityWait). The accumulated wait time
> is reset to 0 whenever a non-local map task is scheduled. That means it takes
> N * wait_time to schedule N non-local map tasks.
> Because of 1, a lot of TT will not be able to take more tasks, even if they
> have free slots. As a result, a lot of the map tasks cannot be scheduled
> locally.
> Because of 2, it's really hard to schedule a non-local task.
> As a result, sometimes we are seeing that it takes more than 2 minutes to
> schedule all the mappers of a job.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
