[
https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dhruba borthakur updated MAPREDUCE-944:
---------------------------------------
Attachment: LoadManager2.txt
Incorporated Matie's review comments.
Vinod: The goal that we have in mind is slightly different from what the
capacity scheduler has done (pl correct me if I am wrong). Unlike the capacity
scheduler, there is no assumption that the user knows (upfront, before
submitting job) how much memory/CPU/network that job will need. A realtime
stream of resource usage will be fed into the new LoadManager that can then
dynamically decide whether to run another task on that machine or not. Your
feedback and experience vis-a-vis that Capacity scheduler is very valuable
here... let's continue the conversation via MAPREDUCE-961
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt, LoadManager2.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate
> allocating new tasks on a particular TaskTracker. In similar lines, it would
> be nice if the FairShare Scheduler can have a pluggable policy to regulate
> new tasks from a particular job. This will allow one to skip scheduling tasks
> of a job that is eating a large percentage of memory in the cluster, i.e.
> fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
