yashmayya opened a new pull request, #17692:
URL: https://github.com/apache/pinot/pull/17692
### Summary
When a query using window funnel aggregation functions (`funnelMaxStep`,
`funnelCompleteCount`, `funnelMatchStep`) times out or is cancelled, the worker
threads spawned by `IndexedTable.finish()` for the multi-threaded final reduce
phase continue running indefinitely. Although `future.cancel(true)` is
correctly called on the futures, the underlying `extractFinalResult()`
computations are tight CPU-bound loops that never check the thread interrupt
flag, so the cancellation has no effect. Repeated timed-out queries compound
this, eventually saturating the thread pool and pegging server CPU until a
restart.
### Why this was missed
Most aggregation functions have trivial `extractFinalResult()`
implementations — `SUM` returns the accumulated value, `COUNT` returns a long,
`AVG` does a division, and sketch-based functions like HLL just call
`estimate()`. These
complete in O(1) and are never a cancellation concern.
The funnel window functions are unique: they defer the actual computation to
`extractFinalResult()`, which performs a
sliding-window pattern match over the full `PriorityQueue` of raw events
accumulated during the aggregate/merge phases.
This is the same reason they're the only functions listed in
`IndexedTable.containsExpensiveAggregationFunctions()` to
trigger multi-threaded execution — but the corresponding
cancellation-awareness was never added to the computation
itself.
### Fix
Add `QueryThreadContext.checkTerminationAndSampleUsagePeriodically()` calls
inside every hot loop in the funnel window functions:
- **`FunnelBaseAggregationFunction.fillWindow()`** — the step-0 seeking
loop (can drain millions of non-matching events)
and the window-filling loop (can move millions of events into the sliding
window when the window size is large)
- **`FunnelMaxStepAggregationFunction.processWindow()`** — iterates the
entire sliding window
- **`FunnelMatchStepAggregationFunction.processWindow()`** — same
- **`FunnelCompleteCountAggregationFunction.extractFinalResult()`** —
inline sliding window iteration
These checks detect three cancellation signals: the `TerminationException`
set by `QueryExecutionContext.terminate()`,
the thread interrupt flag set by `future.cancel(true)`, and deadline
expiration. Any of these causes the worker thread
to throw and unwind immediately.
### Overhead
`checkTerminationAndSampleUsagePeriodically` uses a bitmask (`& 0x1FFF`)
so the actual check only fires every 8,192 loop
iterations. The remaining iterations reduce to a single integer AND +
branch prediction hit. When the check does fire,
it reads one volatile field, calls `Thread.interrupted()`, and compares
`System.currentTimeMillis()` against the
deadline — all sub-microsecond operations, negligible relative to the
per-event `PriorityQueue.poll()` (O(log N)) and
sliding window processing work done in each iteration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
