gortiz opened a new pull request, #19589: URL: https://github.com/apache/pinot/pull/19589
**Draft, not for merge.** This is a spike backing the discussion on #19339, published so the numbers below can be reproduced and argued with. It is **stacked on #19408**. GitHub can only base a PR on a branch in this repository, so the diff here also contains that PR's commits. Only the last two are new: - `SPIKE: streaming restriction kernel (option 4)` - `SPIKE: benchmark streaming against eager restriction` ## What it does #19408 pushes a restriction into an OR by **materializing**: the enclosing AND hands its matched document ids to the subtree, which returns a bitmap. That fixes the reported bug but destroys early termination, which is why it ships disabled and why `AUTO` excludes queries that can stop before draining the filter. This spike delivers the same restriction **as a stream** instead: - `ScanBasedDocIdIterator#matchDocIds(int[], int)` — a re-entrant per-chunk kernel. Unlike `applyAnd` it does not consume the iterator, so it can be called once per chunk. `SVScanDocIdIterator` delegates to the in-place compaction its `ValueMatcher` already does; `MVScanDocIdIterator` loops `doesValueMatch`; everything else gets a default built on `applyAnd`. - `RestrictedScanDocIdIterator` — drives a scan from an upstream candidate stream a chunk at a time, so the scan never sees a document outside the candidate set and the result is never materialized. The only thing that made `applyAnd` non-re-entrant was the trailing `close()` in `SVScanDocIdIterator`, which releases one `ForwardIndexReaderContext`. `MVScanDocIdIterator#applyAnd` was already re-entrant. **It is not wired into `AndDocIdSet`.** The benchmark assembles the streaming tree by hand from the same identities the push-down applies, and asserts at setup that all three strategies return identical documents. ## Numbers `BenchmarkAndRestrictionPushdown`, 2M documents, real `SVScanDocIdIterator` over a fixed-bit forward index, chunk size 256. Time and allocation as a ratio of master's behaviour — lower is better, **1.00x is master**: | shape | consume | sel | eager time | streaming time | eager alloc | streaming alloc | |---|---|---|---|---|---|---| | SCAN_IN_OR | DRAIN | 0.01 | 0.03x | 0.03x | 0.10x | 0.06x | | SCAN_IN_OR | DRAIN | 0.5 | **0.66x** | 1.21x | 1.14x | 0.27x | | SCAN_IN_OR | LIMIT | 0.01 | 0.02x | **0.01x** | 0.10x | 0.06x | | SCAN_IN_OR | LIMIT | 0.5 | 0.58x | **0.04x** | 1.14x | 0.26x | | SCAN_ONLY_OR | DRAIN | 0.01 | **0.32x** | 0.40x | 21.97x | 1.41x | | SCAN_ONLY_OR | DRAIN | 0.5 | **0.34x** | 0.53x | 232.27x | 1.17x | | SCAN_ONLY_OR | LIMIT | 0.01 | 138.85x | **2.23x** | 25.04x | 1.33x | | SCAN_ONLY_OR | LIMIT | 0.5 | 7925.64x | **3.27x** | 383.31x | 1.33x | `SCAN_IN_OR` is a scan next to an index-based predicate inside an OR branch, the shape reported in #19339. `SCAN_ONLY_OR` is scans directly under the OR with no index-based sibling, so the OR is already lazy on master and a LIMIT really does stop early. ## Reading **Streaming removes the cliff.** The worst case goes from 7926x slower to 3.3x, and allocation from 383x to 1.3x. It is also much better than eager wherever the consumer stops early, because it keeps the early termination eager destroys. **Eager is still better for a full drain** — 0.66x against 1.21x on `SCAN_IN_OR` at high candidate density, where streaming is actually slower than master. Chunking costs more than it saves when every matching document is going to be read anyway. So streaming does not remove the need for a mode. It changes what the mode chooses: eager when the query will drain the filter, streaming when it may stop early. `AUTO` stops being a safety gate and becomes a strategy selector, and its "no" branch goes from merely safe to fast. ## What is not measured - The scan is an `SVScanDocIdIterator`. An `ExpressionScanDocIdIterator` — the `IN_SUBQUERY` case that motivated #19339 — builds a whole `ProjectionOperator` per call, so per chunk it will want a much larger chunk, and a larger chunk erodes the early-termination granularity this exists for. That trade-off is the entry criterion for doing this properly and is still open. - Chunk size is fixed at 256; its sensitivity is untested. - Nothing here is wired into the execution path, so there is no end-to-end query measurement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
