Zoltán Borók-Nagy created IMPALA-15657:
------------------------------------------
Summary: CTE producers and consumers can deadlock when a UNION
opens consumers one at a time; LocalExchanger::Pull() ignores cancellation
Key: IMPALA-15657
URL: https://issues.apache.org/jira/browse/IMPALA-15657
Project: IMPALA
Issue Type: Bug
Components: Backend
Reporter: Zoltán Borók-Nagy
CTEProducerNode::GetNext() returns eos only after every consumer has read the
whole exchanger (LocalExchanger::ReadFinished()). UnionNode reads its children
one after another. When the consumers of one CTE sit in later UNION children,
behind a child that itself depends on another producer of the same query, every
producer waits for consumers that are never reached.
*Seen with* a TPC-DS q67a variant that computes each rollup level from the
previous one: CTEs r1..r8, each consumed by the next level and by the UNION ALL
(PLANNER=CALCITE, CTE_THRESHOLD=1, MT_DOP=8, SF3000).
* The first run hung for 24 minutes until it was cancelled.
* The second run finished in 137 s, so the outcome depends on timing.
A stack dump during the hang shows 8 producer threads in
CTEProducerNode::GetNext() / LocalExchanger::ReadFinished(), and the remaining
fragment instance threads in UnionNode::GetNext().
After the cancel, one executor still held the query's memory 180 s later.
LocalExchanger::Pull() waits on batch_available_cv_ without checking for
cancellation, so a consumer whose producer never closes stays blocked.
*Possible fixes:*
* Let the producer return eos as soon as its input is exhausted, instead of
waiting for the readers. This needs the exchanger to outlive the producer's
fragment instance (see IMPALA-15313).
* Make Pull() cancellation-aware, e.g. a timed wait that checks
RuntimeState::is_cancelled().
* Have the planner reject plans where a CTE's consumers are ordered behind a
dependency on that CTE.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
