[
https://issues.apache.org/jira/browse/MAPREDUCE-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12752105#action_12752105
]
Vinod K V commented on MAPREDUCE-944:
-------------------------------------
I see in the patch attached that only one concrete implementation
CapBasedLoadManager is done for the LoadManager which in turn doesn't take into
account any resource usage. I guess you are planning a proper implementation
for this feature regarding fair-share of memory usage in another JIRA.
Some points still not dealt with in this JIRA. I bring about these points so as
to know if you are thinking or have already thought anything about this.
- Job configuration about how users specify the resource usage. Some memory
related configuration properties are added to the framework while working for
memory monitoring on TTs as well as memory usage based scheduling in
CapacityTaskScheduler. You may want to reuse some/all of it.
- Capturing the scheduling decisions involved when we are not able to find a
task from a Schedulable because of lack of resources on a given TaskTasker.
Regarding the latter, the current patch just returns null, which is similar to
the decision CapacityTaskScheduler used to take in previous versions - i.e.
block the TT till it can be given a task from the job at the head of the
queue/pool. Sometime back, we investigated how this approach works with
FairScheduler and realized some important implications. For e.g, because the
order of jobs might change significantly in consecutive iterations of
FairScheduler, just returning null may not work at all. Eventually we may end
up waiting for a long time if significant number of jobs ask for high amount of
resources.
Thoughts?
> Extend FairShare scheduler to fair-share memory usage in the cluster
> --------------------------------------------------------------------
>
> Key: MAPREDUCE-944
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-944
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: dhruba borthakur
> Attachments: LoadManager.txt
>
>
> The FairShare Scheduler has an extensible LoadManager API to regulate
> allocating new tasks on a particular TaskTracker. In similar lines, it would
> be nice if the FairShare Scheduler can have a pluggable policy to regulate
> new tasks from a particular job. This will allow one to skip scheduling tasks
> of a job that is eating a large percentage of memory in the cluster, i.e.
> fair-share of memory resources among jobs.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
