speculative execution is too aggressive under certain conditions
----------------------------------------------------------------
Key: MAPREDUCE-2062
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2062
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Reporter: Joydeep Sen Sarma
The function canBeSpeculated has subtle bugs that cause too much speculation in
certain cases.
- it compares the current progress of the task with the last observed mean of
all the tasks. if only one task is in question - then the progress rate decays
as time progresses (in the absence of updates) and std-dev is zero. So a job
with a single reducer or mapper is almost always speculated.
- is only a single task has reported progress - then the stddev is zero. so
other tasks may be speculated aggressively.
- several tasks take a while to report progress initially. they seem to get
speculated as soon as speculative-lag is over. the lag should be configurable
at the minimum.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
