[
https://issues.apache.org/jira/browse/MAPREDUCE-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742989#action_12742989
]
eric baldeschwieler commented on MAPREDUCE-718:
-----------------------------------------------
This will possibly put a lot more load on the JT. Are we confident this will
not impact performance?
Down the road...
A thought would be to have a statistics proxy for each job. It could collect
statistics, record them for later user us and pass speculative decisions up to
the JT. A possible step towards moving all the map-reduce logic out of the JT
into a per job master process, which would have lots of nice properties.
> Support for per-phase speculative execution
> -------------------------------------------
>
> Key: MAPREDUCE-718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Devaraj Das
> Fix For: 0.21.0
>
>
> It would be good to have support for per-phase speculative execution where
> the algorithm looks at the current phase of a task, and compares with the
> other tasks in the same phase before deciding to launch a speculative task.
> That would have the following benefits:
> 1) Support for jobs where map tasks progresses jumps from 0% to 100%. This is
> true for some jobs like randomwriter. Today, we would launch speculative
> tasks for such jobs (assuming that the tasks are not making progress). But
> most of them would be unnecessary.
> 2) In reality, for reduces, the three phases are quite different from each
> other, and they take different times too. We should see better results when
> we look at per-phase speculation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
