Akanksha-kedia opened a new pull request, #19092: URL: https://github.com/apache/pinot/pull/19092
## Description `DistinctCombineOperator` previously followed the generic `BaseSingleBlockCombineOperator` queue-based pattern: each segment operator posted a `DistinctResultsBlock` to a `LinkedBlockingQueue`, and the main thread serially dequeued and merged them — one `DistinctTable` allocation and one queue round-trip per segment. This PR replaces that with **per-task thread-local accumulation**, following the same design already used by `GroupByCombineOperator`: ### Changes **`DistinctCombineOperator`** - Each worker task claims a unique slot `[0, numTasks)` via `AtomicInteger` and merges every segment it processes directly into that slot's `DistinctTable` — no queue, no per-segment allocation. - After all tasks finish (signaled by `CountDownLatch`), the main thread merges at most `numTasks` per-task tables rather than `numSegments` tables. - Early termination sets `_nextOperatorId` to `_numOperators` as soon as a task's accumulated table satisfies the LIMIT, stopping all tasks. - `CountDownLatch.countDown()→await()` establishes the happens-before relationship that makes per-task table writes visible to the main thread (same guarantee used by `GroupByCombineOperator`). **`DistinctCombineOperatorTest`** (new) - Full scan: verifies all `numSegments × uniquePerSegment` distinct values collected - Early termination: verifies fewer docs scanned when LIMIT is small - ORDER BY ASC/DESC: verifies correct sorted output - Cross-segment deduplication: verifies overlapping values across segments are merged correctly ### Performance impact For a `SELECT DISTINCT col LIMIT L` query over S segments with T worker threads (T << S): - **Before**: S `DistinctTable` allocations + S queue operations + S serial merges - **After**: T `DistinctTable` allocations + T final merges The benefit is most pronounced for queries with a small `LIMIT` over many segments, where the bounded table size makes allocation cost non-trivial. ## Test plan - [ ] `DistinctCombineOperatorTest` — 5 unit tests, all passing - [ ] Existing `DistinctQueriesTest` integration tests pass -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
