Denys Kuzmenko created HIVE-30207:
-------------------------------------
Summary: LLAP: a fragment waiting for a queued fragment's output
holds its executor until a source state update arrives
Key: HIVE-30207
URL: https://issues.apache.org/jira/browse/HIVE-30207
Project: Hive
Issue Type: Bug
Reporter: Denys Kuzmenko
An LLAP daemon preempts a running fragment that cannot finish in favour of a
queued fragment that can. A fragment's finishable state only changes when a
source state update for the query reaches the daemon
(ContainerRunnerImpl.sourceStateUpdated -> QueryInfo.sourceStateUpdated -> the
finishable callbacks): it is edge triggered, and nothing re-reads or reconciles
it. TaskExecutorService considers a kill only when the head of the wait queue
is finishable or guaranteed.
So if a daemon misses one update, a running fragment that waits for the output
of a queued fragment keeps its executor for good. Seen on TPC-DS q14 at 10 TB,
24 daemons x 12 executors: every executor held a Map 19 fragment waiting for
the broadcast of Reducer 25, and the last four Reducer 25 fragments were queued
on one daemon. When Reducer 29 succeeded, the other 23 daemons preempted a Map
19 fragment within 2 ms; that daemon never did, and the query hung for 63
minutes. The AM sees the queued attempts in the daemon's heartbeat, so the Tez
attempt timeout never fires. Why that daemon missed the update is a separate
question (the AM's SourceStateTracker records a node as updated
before the request is delivered and never re-sends); the scheduler should not
depend on it.
Proposal: a queued fragment may preempt a running fragment of the same query
that reads an input of the queued fragment's vertex, regardless of the
finishable state, since the running fragment cannot finish before the queued
one runs. This mirrors the AM's rule of preempting only tasks below the
requesting one in the DAG.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
