[
https://issues.apache.org/jira/browse/MAPREDUCE-5019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13615072#comment-13615072
]
Damien Hardy commented on MAPREDUCE-5019:
-----------------------------------------
I just discover the option :
mapred.reduce.slowstart.completed.maps (default is 0.05)
that could be used to prevent big job from staling smaller ones waiting for
reducer slots.
Trying with a value of 0.8 (80% of mappers have to be terminated before
starting reducers)
> Fair scheduler should allow peremption on reducer only
> ------------------------------------------------------
>
> Key: MAPREDUCE-5019
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5019
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv1, scheduler
> Affects Versions: 2.0.2-alpha
> Environment: CDH4.1.2
> Reporter: Damien Hardy
> Priority: Minor
> Labels: scheduler, scheduling
>
> Fair scheduler is very good.
> But having a big MR job running lots of mapper and reducer( 10M + 10R )
> Then a small MR on the same pool (1M + 1R)
> having slots for 10 mapper and 10 reducer
> - The big job take all the map slots
> - The small job wait for a map slot
> - 1rst big job map task finish
> - the small job take the map slot it needs
> - meanwhile all the reducer of the big job take all the reducer slot to copy
> and sort
> - the small job end is map and wait for the all maps to end and for 1
> reducer to end before accessing for a reducer slot.
> - all the reducer stalled after sorting waiting for the mapper to end one
> by one...
> If I have a big job and a lot of small, I don't want new small arriving and
> killing running map tasks of big job to get a slot.
> I think it could be useful that the small job can kill a reducer tasks (and
> only reducer) to end before the big job finish all its map tasks and a
> reducer.
> rules can be : a job having all its map finished and waiting for reducer slot
> can kill reducer tasks from a job that still have map slot running (assuming
> they are just waiting for copy and sort)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
