Shuffle should be refactored to a separate task by itself
---------------------------------------------------------
Key: HADOOP-1339
URL: https://issues.apache.org/jira/browse/HADOOP-1339
Project: Hadoop
Issue Type: New Feature
Components: mapred
Reporter: Devaraj Das
Fix For: 0.14.0
Currently, shuffle phase is part of the reduce task. The idea here is to move
out the shuffle as a first-class task. This will improve the usage of the
network since we will then be able to schedule shuffle tasks independently, and
later on pin reduce tasks to those nodes. This will make most sense for apps
where there are multiple waves of reduces (the second wave of reduces can
directly start off doing the "reducer" phase).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
