Allow different slowTaskThreshold for mappers and reducers
----------------------------------------------------------
Key: MAPREDUCE-1903
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1903
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Affects Versions: 0.22.0
Reporter: Scott Chen
Assignee: Scott Chen
Fix For: 0.22.0
We have been running the new speculative logic in HADOOP-2141 done by Andy in
our production cluster.
One thing that we observed is that there are significantly low number of
speculative reducers.
We have seen usually 100 speculative mappers launched per minute but
speculative reducers are usually less than 5.
But reducers are usually where we get complains about having speculative issues.
These two types of tasks has different properties and different needs.
It would be nice if we can configure the slow threshold separately to deal with
them separately.
We can add the following config keys to allow setting them independently.
MRJobConfig.SPECULATIVE_MAP_SLOWTASK_THRESHOLD
MRJobConfig.SPECULATIVE_REDUCE_SLOWTASK_THRESHOLD
Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
