Jackie-Jiang commented on issue #19339: URL: https://github.com/apache/pinot/issues/19339#issuecomment-5593130904
@gortiz Some historical context that seems relevant to #19408: - #10396 describes essentially the same missing candidate-bitmap pushdown through an OR: selective indexed predicates combined with `OR(isSubnetOf(...), ...)`, where `advance()` can scan almost the entire segment. It proposes wiring the outer bitmap into the nested scans. I previously linked it to #9839, which tracks restricted scans, switching from indexes to scans when few candidates remain, and lazy evaluation / limit pushdown. - #9402 is an earlier production report of `P AND ((A AND B) OR C)`, with roughly 42.8M versus 554 filter entries scanned after changing a nested predicate. The discussion also points to #7597 for passing previously matched documents into the range index. - #5833, addressing #5596, experimented with changing how multiple scan predicates are evaluated against bitmap candidates. It was closed without merging after benchmarks showed no benefit with enough appropriate indexes and mixed gains/losses with fewer indexes ([author's explanation](https://github.com/apache/pinot/pull/5833#issuecomment-721393089)). This was a different scan strategy, so it is a caution about workload dependence rather than evidence against OR candidate pushdown itself. - #12611 accidentally stopped registering bitmap iterators in `OrDocIdSet`; #15756 fixed the resulting loss of bitmap-based scan restriction. One workload saw almost 100x higher latency in Pinot 1.2 ([report](https://github.com/apache/pinot/pull/15756#discussion_r2081982375)), and some combinations of sorted and inverted indexes also returned incorrect results. This was an implementation regression, but it illustrates how sensitive the surrounding iterator contracts are. - In #8453, we already discussed that applying a bitmap eagerly can be appropriate for aggregations while selections should evaluate lazily and stop at their limit ([discussion](https://github.com/apache/pinot/issues/8453#issuecomment-1085194696)). This is directly relevant to the tradeoff measured in #19408. My reading is that the history supports fixing the execution-layer gap. It does not establish that an earlier implementation of this same OR pushdown was tried and failed. Restricting which documents a filter examines and deciding how eagerly to examine them are separate decisions; improving the former can still regress early-terminating queries through the latter. The disabled default and conservative `AUTO` mode in #19408 acknowledge that tradeoff. Could you evaluate the history above against the current proposal and see whether any adjustments are needed? In particular: 1. Can candidate propagation preserve lazy/batched evaluation and early termination, or should that remain an explicit follow-up with the current mode gating? 2. Should deferral be limited to subtrees that contain scan/expression predicates, given the acknowledged extra bitmap work for entirely indexed subtrees? 3. Can we validate with latency and allocation benchmarks, in addition to entries scanned, across aggregations and limited selections, sparse and dense candidate sets, and mixed versus entirely indexed branches? Please summarize which historical cases are already covered and whether they suggest changes to the implementation, mode eligibility, or regression tests. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
