Ahmed Hussein created TEZ-4067:
----------------------------------
Summary: Tez Speculation decision is calculated on each update by
the dispatcher
Key: TEZ-4067
URL: https://issues.apache.org/jira/browse/TEZ-4067
Project: Apache Tez
Issue Type: Improvement
Reporter: Ahmed Hussein
LegacySpeculator is an object field in VertexImpl. Therefore, all events are
handled synchronously by the caller (dispatcher). This implies the following:
# the dispatcher spends long time executing updateStatus as it needs to check
the runtime estimation of the tezAttempts within the vertex.
# the speculator is per stage: lunching a speculation may not the optimum
decision. Ideally, based on resources, speculated tasks should be the ones with
slowest progress.
# the time between speculation is skewed because there is a big delay for the
dispatcher to complete a full cycle. Also, speculation will be more aggressive
compared to MR because MR waits for "soonest.retry.after.speculate" whenever a
task is speculated. On the other hand, Tez speculates more tasks as it
processes stages in parallel.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
