[
https://issues.apache.org/jira/browse/HADOOP-2141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12566194#action_12566194
]
Joydeep Sen Sarma commented on HADOOP-2141:
-------------------------------------------
just wanted to leave behind some recent experiences. we have started running
with gap=0.9 and only turned on spec. for maps. this is working out pretty
well. i see very few spec. tasks - but they do show up when things are stuck -
which is exactly what we want.
the only remaining flaw is that hasSpeculative() should not be using 'progress'
- but rather 'progress/running time'. the rationale is that a late task is
always going to have low progress (even after the lag time) - so what matters
is progress made relative to the run-time. if on average tasks require 5
minutes to make 20% progress then making 5% progress in 1 minute is perfectly
fine (but today may cause spec. execution even for our conservative setting).
> speculative execution start up condition based on completion time
> -----------------------------------------------------------------
>
> Key: HADOOP-2141
> URL: https://issues.apache.org/jira/browse/HADOOP-2141
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.15.0
> Reporter: Koji Noguchi
> Assignee: Arun C Murthy
> Fix For: 0.17.0
>
>
> We had one job with speculative execution hang.
> 4 reduce tasks were stuck with 95% completion because of a bad disk.
> Devaraj pointed out
> bq . One of the conditions that must be met for launching a speculative
> instance of a task is that it must be at least 20% behind the average
> progress, and this is not true here.
> It would be nice if speculative execution also starts up when tasks stop
> making progress.
> Devaraj suggested
> bq. Maybe, we should introduce a condition for average completion time for
> tasks in the speculative execution check.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
