[
https://issues.apache.org/jira/browse/TEZ-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14180286#comment-14180286
]
Bikas Saha commented on TEZ-14:
-------------------------------
You are right. In the long term there may be cases where some customization may
be needed. This could either be done via configurable speculator or by
involving the vertex manager since the vertex manager does the original task
scheduling. However, there are no concrete cases now. Also, speculators would
need stats etc from the system and we dont yet have a stats API that exposes
this data. Until then we might be better off not making that configurable and
using this basic speculator that works for the straightforward cases. That is
why its currently not configurable.
Also, once some of the stats work is done, I plan to replace this legacy
speculator with a more advanced version that understands source slowness, does
pro-active cancellation etc like described in the Mantri paper above. My guess
is that a well-written default speculator will remove the need for custom
speculators.
Did you get a chance to review the code or try it out on a cluster?
> Support for speculation of slow tasks
> -------------------------------------
>
> Key: TEZ-14
> URL: https://issues.apache.org/jira/browse/TEZ-14
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-14.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
