[
https://issues.apache.org/jira/browse/MAPREDUCE-1463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12830502#action_12830502
]
Amar Kamat commented on MAPREDUCE-1463:
---------------------------------------
Scott,
- How do you define small jobs. Shouldnt it be based on total number of tasks
instead of considering maps and reduces individually?
- Why do we need special case for small jobs? If its for fairness then this
piece of code rightly belongs to contrib/fairscheduler, no?
- If not for fairness then what is the problem with the current framework w.r.t
small jobs?
- Can be fixed by simple (configuration-like) tweaking?
- If not then whats the right fix.
Wouldn't the reducers be scheduled faster if
'mapreduce.job.reduce.slowstart.completedmaps' is set to 0? If not then can we
change the slowstart feature to get it right?
> Reducer should start faster for smaller jobs
> --------------------------------------------
>
> Key: MAPREDUCE-1463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1463
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: contrib/fair-share
> Reporter: Scott Chen
> Assignee: Scott Chen
> Attachments: MAPREDUCE-1463-v1.patch, MAPREDUCE-1463-v2.patch
>
>
> Our users often complain about the slowness of smaller ad-hoc jobs.
> The overhead to wait for the reducers to start in this case is significant.
> It will be good if we can start the reducer sooner in this case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
