[ https://issues.apache.org/jira/browse/MAPREDUCE-5844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14033298#comment-14033298 ]
Hadoop QA commented on MAPREDUCE-5844: -------------------------------------- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650692/MAPREDUCE-5844.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4663//testReport/ Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4663//console This message is automatically generated. > Reducer Preemption is too aggressive > ------------------------------------ > > Key: MAPREDUCE-5844 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5844 > Project: Hadoop Map/Reduce > Issue Type: Bug > Reporter: Maysam Yabandeh > Assignee: Maysam Yabandeh > Attachments: MAPREDUCE-5844.patch, MAPREDUCE-5844.patch, > MAPREDUCE-5844.patch > > > We observed cases where the reducer preemption makes the job finish much > later, and the preemption does not seem to be necessary since after > preemption both the preempted reducer and the mapper are assigned > immediately--meaning that there was already enough space for the mapper. > The logic for triggering preemption is at > RMContainerAllocator::preemptReducesIfNeeded > The preemption is triggered if the following is true: > {code} > headroom + am * |m| + pr * |r| < mapResourceRequest > {code} > where am: number of assigned mappers, |m| is mapper size, pr is number of > reducers being preempted, and |r| is the reducer size. > The original idea apparently was that if headroom is not big enough for the > new mapper requests, reducers should be preempted. This would work if the job > is alone in the cluster. Once we have queues, the headroom calculation > becomes more complicated and it would require a separate headroom calculation > per queue/job. > So, as a result headroom variable is kind of given up currently: *headroom is > always set to 0* What this implies to the speculation is that speculation > becomes very aggressive, not considering whether there is enough space for > the mappers or not. -- This message was sent by Atlassian JIRA (v6.2#6252)