[
https://issues.apache.org/jira/browse/MAPREDUCE-718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742766#action_12742766
]
Devaraj Das commented on MAPREDUCE-718:
---------------------------------------
The current speculative execution heuristic assumes inputs are homogeneous.
This is not always true. Inputs are often skewed in practical cases. AS of
today, we might end up launching speculative tasks for the ones that are
processing large inputs. We should base the criteria of choosing what task to
speculate on something that reflects the actual progress and that doesn't get
affected by the skewness in the input.
One thought there is to base the choice of speculative candidates on the rate
of increase of the relevant System/Framework Counters in each task phase. For
example, for tasks in the map phase, we can monitor the rate of increase of
MAP_INPUT_RECORDS. For reduce tasks in the shuffle phase, we look at
SHUFFLE_BYTES counter and so on. We also need to maintain the per-phase
statistics for the counters, and consider the current phase of a given task for
considering it for speculation.
Today, the task counters is transmitted once a minute from the tasktrackers. I
propose that for the system counters we send the counters in every heartbeat.
The communication of the system counters information can be made efficient by
having a customized serialization for them. Since the system counter names are
known to both the JobTracker and the TaskTrackers, the order in which they are
serialized/deserialized is a good enough indicator of which particular counter
we are talking about. Also, the counter values can be serialized as vInts.
@Zhang, for speculative execution, we don't migrate tasks.
> Support for per-phase speculative execution
> -------------------------------------------
>
> Key: MAPREDUCE-718
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-718
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Affects Versions: 0.21.0
> Reporter: Devaraj Das
> Fix For: 0.21.0
>
>
> It would be good to have support for per-phase speculative execution where
> the algorithm looks at the current phase of a task, and compares with the
> other tasks in the same phase before deciding to launch a speculative task.
> That would have the following benefits:
> 1) Support for jobs where map tasks progresses jumps from 0% to 100%. This is
> true for some jobs like randomwriter. Today, we would launch speculative
> tasks for such jobs (assuming that the tasks are not making progress). But
> most of them would be unnecessary.
> 2) In reality, for reduces, the three phases are quite different from each
> other, and they take different times too. We should see better results when
> we look at per-phase speculation.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
