[
https://issues.apache.org/jira/browse/TEZ-14?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14167844#comment-14167844
]
Bikas Saha commented on TEZ-14:
-------------------------------
I have attached an initial port of the MR default speculator to Tez
Overall notes
1) Ported the DefaultSpeculator and estimator code to Tez and made some changes
that made sense for Tez. Eg. remove container need tracking for M and R tasks
because Tez has generic priority based ordering and scheduling. [~jlowe]
[~jeagles] Could you please take a close look at this code.
2) Speculator runs per vertex and gets status updates from task attempts.
3) State machines already had support for speculation and needed only minor bug
fixes. [~jlowe][~jeagles] Do you know of any state machine fixes that were made
in MR for speculation recently?
Improved MockDAGAppMaster infra to add some basic test cases for useful and
non-useful speculation.
The speculator is not pluggable etc because this is mainly targeted to match
legacy MR speculation. A more advanced speculator would need much more
information and stats that are currently not readily available.
TBD - more tests, running speculative tasks at lower priority than normal tasks
so that they dont block normal tasks of the same vertex, counters/ATS-events,
performance measurements/heuristics to check if this is safe to run on the
AsyncDispatcher thread. [~jlowe][~jeagles] Do you have some idea of the
performance of this code in Yahoo clusters?
[~hitesh] [~jeagles][~jlowe] Please review.
> Support for speculation of slow tasks
> -------------------------------------
>
> Key: TEZ-14
> URL: https://issues.apache.org/jira/browse/TEZ-14
> Project: Apache Tez
> Issue Type: Improvement
> Reporter: Bikas Saha
> Assignee: Bikas Saha
> Attachments: TEZ-14.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
