Launch reduces only after a few maps have run in the Fair Scheduler
-------------------------------------------------------------------
Key: HADOOP-4666
URL: https://issues.apache.org/jira/browse/HADOOP-4666
Project: Hadoop Core
Issue Type: New Feature
Components: contrib/fair-share
Reporter: Matei Zaharia
It makes no sense to schedule reduces for a job before its maps have started
running. As an initial fix, we will wait until a certain percent have run
(likely 10%). In the future it would be good to choose the time to wait based
on amount of map output data as well - launching reducers that will mostly be
idle is not helpful. Average amount of map output bytes per mapper is easy to
compute using counters in JobInProgress.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
