Optimize finding of speculative tasks
-------------------------------------
Key: HADOOP-2874
URL: https://issues.apache.org/jira/browse/HADOOP-2874
Project: Hadoop Core
Issue Type: Improvement
Components: mapred
Reporter: Amar Kamat
Assuming HADOOP-2119 provides better data structures for handling running TIPs,
finding new speculative tasks can be further optimized. Two of which could be
1) {{conf.getMapSpeculativeExecution()}} and
{{conf.getReduceSpeculativeExecution()}} should be moved to {{JobInProgress}}.
A simple check for this boolean can prove useful before checking for
speculative tasks. This will be useful for jobs with large maps and reducers
where scanning all the TIPs can be costly.
2) Since the progress of a TIP changes only when
{{TaskInProgress.recomputeProgress()}} is invoked, it makes more sense to check
for speculation in {{JobInProgress.updateTaskStatus()}} and move the TIPs that
can be speculated to the front of the running queue.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
