Zoltán Borók-Nagy created IMPALA-15655:
------------------------------------------
Summary: Coordinator crash or query hang when CTE materialization
is combined with COMPUTE_PROCESSING_COST
Key: IMPALA-15655
URL: https://issues.apache.org/jira/browse/IMPALA-15655
Project: IMPALA
Issue Type: Bug
Components: Frontend, Backend
Reporter: Zoltán Borók-Nagy
On master 9a6a6b1432, TPC-DS q18 (SF10000, 12 executors) crashes the
coordinator or hangs with CTE materialization plus processing-cost-based
parallelism:
{noformat}
set MT_DOP=8;
set PLANNER=CALCITE;
set COMPUTE_PROCESSING_COST=true;
set PROCESSING_COST_MIN_THREADS=8;
set CTE_THRESHOLD=1;
{noformat}
These options crash the coordinator during admission. Adding
MAX_FRAGMENT_INSTANCES_PER_NODE=16 gives a hang instead: all rows are fetched
after 32 s, but close never returns. Adding BROADCAST_TO_PARTITION_FACTOR and
BROADCAST_BYTES_LIMIT still hangs; adding RUNTIME_FILTER_MAX_SIZE and
EXEC_TIME_LIMIT_S on top crashes again (reproduced twice). Without
CTE_THRESHOLD=1 the query runs fine: 18 s with
MAX_FRAGMENT_INSTANCES_PER_NODE=16, also with the broadcast or runtime filter
options.
*Crash:*
{noformat}
SIGSEGV (SEGV_MAPERR at 0x6d3d24670)
impala::Scheduler::ComputeFragmentExecParams
impala::Scheduler::Schedule
impala::AdmissionController::ComputeGroupScheduleStates
{noformat}
The faulting access is the map update in the "Set destinations,
per_exch_num_senders, sender_id" loop:
{code}
(*dest_state->exec_params->mutable_per_exch_num_senders())[exch_id] += ...
{code}
dest_state->exec_params is a garbage pointer. GetFragmentIdx() and
GetFragmentScheduleState() index plain vectors without bounds checks in release
builds. Hypothesis: an out-of-range dest_node_id or fragment index for a CTE
fragment. That would explain both the random crash address and the hang:
depending on the other options, the invalid read either faults or yields a
wrong consumer count.
*Hang:* the query stays FINISHED at 100% until cancelled. Two CTE producer
instances never complete. A minidump shows one in LocalExchanger::Release()
under CTEProducerNode::Close (cte-producer-node.cc:135), which waits until
consumers_done_ == consumer_count_. The count comes from the scheduler's
num_cte_consumers. If it does not match the consumer instances that call
CloseConsumer(), the producer waits forever. Cancellation does not wake it.
*Proposal:*
* Reproduce on a debug build, which should hit the DCHECKs and pinpoint the bad
index.
* Check the scheduler's CTE producer/consumer linking (num_cte_consumers) under
COMPUTE_PROCESSING_COST.
* Make cancellation wake a producer blocked in LocalExchanger::Release().
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
