[
https://issues.apache.org/jira/browse/MAPREDUCE-5643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tang shanjiang updated MAPREDUCE-5643:
--------------------------------------
Status: Open (was: Patch Available)
> DynamicMR: A Dynamic Slot Utilization Optimization Framework for Hadoop MRv1
> ----------------------------------------------------------------------------
>
> Key: MAPREDUCE-5643
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5643
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Affects Versions: 1.2.1
> Reporter: tang shanjiang
> Assignee: tang shanjiang
> Labels: performance
>
> Hadoop MRv1 uses the slot-based resource model with the static configuration
> of map/reduce slots in advance. Due to the rigid execution order between map
> and reduce tasks in a MapReduce environment and the strict execution
> constrain that map tasks can only run map slots and reduce tasks can only
> reduce slots, slots can be severely under-utilized, which significantly
> degrades the performance.
> In contrast to YARN that gives up the slot-based resource model to maximize
> resource utilization, we keep the slot-based model and propose a dynamic slot
> utilization optimization system called DynamicMR to improve the performance
> of Hadoop by maximizing the slots utilization and improving utilization
> efficiency while guaranteeing the fairness across pools. It consists of three
> levels of scheduling components, namely, Dynamic Hadoop Fair Scheduler
> (DHFS), Dynamic Speculative Task Scheduler (DSTS), and Data Locality
> Maximization Scheduler (DLMS).
> Our tests show that DynamicMR outperforms YARN for MapReduce workloads with
> multiple jobs, especially when the number of jobs is large.
--
This message was sent by Atlassian JIRA
(v6.1#6144)
