[
https://issues.apache.org/jira/browse/MAPREDUCE-706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matei Zaharia updated MAPREDUCE-706:
------------------------------------
Attachment: mapreduce-706.v5.patch
Here's a new patch that includes the design docs.
> Support for FIFO pools in the fair scheduler
> --------------------------------------------
>
> Key: MAPREDUCE-706
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-706
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
> Reporter: Matei Zaharia
> Assignee: Matei Zaharia
> Attachments: fsdesigndoc.pdf, fsdesigndoc.tex, mapreduce-706.patch,
> mapreduce-706.v1.patch, mapreduce-706.v2.patch, mapreduce-706.v3.patch,
> mapreduce-706.v4.patch, mapreduce-706.v5.patch
>
>
> The fair scheduler should support making the internal scheduling algorithm
> for some pools be FIFO instead of fair sharing in order to work better for
> batch workloads. FIFO pools will behave exactly like the current default
> scheduler, sorting jobs by priority and then submission time. Pools will have
> their scheduling algorithm set through the pools config file, and it will be
> changeable at runtime.
> To support this feature, I'm also changing the internal logic of the fair
> scheduler to no longer use deficits. Instead, for fair sharing, we will
> assign tasks to the job farthest below its share as a ratio of its share.
> This is easier to combine with other scheduling algorithms and leads to a
> more stable sharing situation, avoiding unfairness issues brought up in
> MAPREDUCE-543 and MAPREDUCE-544 that happen when some jobs have long tasks.
> The new preemption (MAPREDUCE-551) will ensure that critical jobs can gain
> their fair share within a bounded amount of time.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
