[
https://issues.apache.org/jira/browse/MAPREDUCE-1608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12847243#action_12847243
]
Amar Kamat commented on MAPREDUCE-1608:
---------------------------------------
bq. Interesting usage case. Are there other tasks running on the same node as
the lone reduce task?
Speculation makes sense when you compare similar tasks as we can easily rule
out the code/logic differences.
The way I understand this is that you are trying to label/weigh the tasktracker
based on how the currently running tasks are behaving on the given tracker. How
about making a note of how many tasks (of a given type) on a given tracker got
speculated and making scheduling decisions based on that. There might be cases
where all the reducers get speculated on a given tracker. This should result
into not scheduling reduces on such nodes and in future utilize it (i.e all the
slots) completely for maps.
For example:
If a tracker T asks for new tasks of type X from job J, then schedule the task
X of job J on T if
(number of tasks speculated of T / number of tasks scheduled on T) < C (where
C is the threshold and default value = 0.8)
This condition says a simple thing, if the job cares about speculation, the
there is no point in running a task of type X on tracker T as there is higher
chance that it might get speculated. Thoughts?
> Allow users to do speculative execution of a task manually
> ----------------------------------------------------------
>
> Key: MAPREDUCE-1608
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1608
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Reporter: Scott Chen
> Assignee: Scott Chen
>
> Speculative execution improves the latency of the job. Sometimes the job has
> few very slow reducers. Spending a little more resource on speculative tasks
> can improve the latency a lot. It will be nice that the users can manually
> select one task and force the speculative execution on that task just like we
> can manually kill/fail task.
> The proposal is add link says "speculate" in taskdetails.jsp page where we do
> "kill/fail".
> Thoughts?
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
