tang shanjiang created MAPREDUCE-5643:
-----------------------------------------
Summary: DynamicMR: A Dynamic Slot Utilization Optimization
Framework for Hadoop MRv1
Key: MAPREDUCE-5643
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5643
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: contrib/fair-share
Affects Versions: 1.2.1
Reporter: tang shanjiang
Hadoop MRv1 uses the slot-based resource model with the static configuration of
map/reduce slots in advance. Due to the rigid execution order between map and
reduce tasks in a MapReduce environment and the strict execution constrain that
map tasks can only run map slots and reduce tasks can only reduce slots, slots
can be severely under-utilized, which significantly degrades the performance.
In contrast to YARN that gives up the slot-based resource model to maximize
resource utilization, we keep the slot-based model and propose a dynamic slot
utilization optimization system called DynamicMR to improve the performance of
Hadoop by maximizing the slots utilization and improving utilization efficiency
while guaranteeing the fairness across pools. It consists of three levels of
scheduling components, namely, Dynamic Hadoop Fair Scheduler (DHFS), Dynamic
Speculative Task Scheduler (DSTS), and Data Locality Maximization Scheduler
(DLMS).
Our tests show that DynamicMR outperforms YARN for MapReduce workloads with
multiple jobs, especially when the number of jobs is large.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
