Allow TaskScheduler manage number slots on TaskTrackers
-------------------------------------------------------
Key: MAPREDUCE-2108
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2108
Project: Hadoop Map/Reduce
Issue Type: New Feature
Components: contrib/capacity-sched, contrib/fair-share
Affects Versions: 0.22.0
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
Currently the map slots and reduce slots are managed by TaskTracker
configuration.
To change the task tracker slots, we need to restart the TaskTrackers.
Also, for a non-uniform cluster, we have to deploy different sets of
configuration.
Now JobTracker holds the CPU and memory status of TaskTrackers (MAPREDUCE-1218).
So it makes sense to just let JobTracker.taskScheduler decided the number of
slots on each node.
This way we can
1. Change the number of slots dynamically without restarting TaskTracker
2. Use different number of slots based on the resource of a TaskTracker
To achieve this, we need to change the logic that we use totalMapSlots and
totalReduceSlots in JobTracker.
I think they are used in WebUI and speculativeCap.
We will need to make JobTracker calculate these numbers from TaskScheduler and
TaskTrackerStatus.
TaskScheduler and TaskTracker can both hold their maximum slots. We pick the
smaller one.
Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
