[
https://issues.apache.org/jira/browse/MAPREDUCE-2108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964582#action_12964582
]
Scott Chen commented on MAPREDUCE-2108:
---------------------------------------
bq. Can you please describe the changes you are making here...
Sorry for not making this clear. The purpose here is to move the control of
maximum slots to TaskScheduler.
This allows TaskScheduler to perform better resource scheduling and allows
changing the number of slots on fly.
The changes made in the patch are the following:
1. Add a getMaxSlots(TaskTrackerStatus, TaskType) method to TaskScheduler.
2. Replace TaskTrackerStatus.getMaxSlots() everywhere in the JobTracker with
the above method.
This way the JobTracker pulls the "maximum slots" information from
TaskScheduler.
The default method in TaskScheduler.getMaxSlots() is to simply report
TaskTrackerStatus.getMaxSlots(). So it will not change any behavior.
But people can overwrite this method and put more sophisticated logic in it
(See MAPREDUCE-2198).
> Allow TaskScheduler manage number slots on TaskTrackers
> -------------------------------------------------------
>
> Key: MAPREDUCE-2108
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/capacity-sched, contrib/fair-share
> Affects Versions: 0.22.0
> Reporter: Scott Chen
> Assignee: Scott Chen
> Fix For: 0.22.0
>
> Attachments: MAPREDUCE-2108-v2.txt, MAPREDUCE-2108.txt
>
>
> Currently the map slots and reduce slots are managed by TaskTracker
> configuration.
> To change the task tracker slots, we need to restart the TaskTrackers.
> Also, for a non-uniform cluster, we have to deploy different sets of
> configuration.
> Now JobTracker holds the CPU and memory status of TaskTrackers
> (MAPREDUCE-1218).
> So it makes sense to just let JobTracker.taskScheduler decided the number of
> slots on each node.
> This way we can
> 1. Change the number of slots dynamically without restarting TaskTracker
> 2. Use different number of slots based on the resource of a TaskTracker
> To achieve this, we need to change the logic that we use totalMapSlots and
> totalReduceSlots in JobTracker.
> I think they are used in WebUI and speculativeCap.
> We will need to make JobTracker calculate these numbers from TaskScheduler
> and TaskTrackerStatus.
> TaskScheduler and TaskTracker can both hold their maximum slots. We pick the
> smaller one.
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
