xiangfu0 opened a new pull request, #19582: URL: https://github.com/apache/pinot/pull/19582
## Summary StarTree serialized bitmap inputs currently become a per-row `byte[]` and a fully deserialized temporary bitmap before union. Add an opt-in path that consumes each mapped/direct chunk-buffer view synchronously through the forward-index reader, projection and `DISTINCTCOUNTBITMAP` aggregation. Enable with `SET useBufferBackedDistinctCountBitmap = true;` (default false). This applies to projected serialized SV BYTES, including StarTree inputs. Numeric/hash semantics and the existing intermediate/wire format stay unchanged. - Add a default callback API to `ForwardIndexReader`, with byte-array fallback for existing implementations. Native variable-byte readers serve read-only borrowed views; V4 behavior is inherited by V5/V6. Compressed oversized values release temporary direct storage in `finally`. - Do not cache borrowed views. Union consumes them before the next read; the mutable accumulator owns its containers. Cover scalar, SV group-by and MV group-by paths and skip null rows before parsing. - Convert the accumulator to the existing `RoaringBitmap` intermediate type at extraction. This adds a conversion cost and does **not** make the accumulator or broker merge off heap. ## Why draft The path is implemented and correctness tests pass, but it is not ready to recommend enabling. A local mapped-reader → projection → aggregation probe found a sparse-data regression, despite lower allocations on dense input: | Fixture / codec | CPU ms: existing → buffer | Heap allocated MiB: existing → buffer | |---|---:|---:| | sparse-overlap / PASS_THROUGH | 11.53 → 21.25 | 17.963 → 20.007 | | sparse-overlap / LZ4 | 11.19 → 20.94 | 17.963 → 19.985 | | dense-overlap / PASS_THROUGH | 0.43 → 0.66 | 0.537 → 0.069 | | dense-overlap / LZ4 | 0.42 → 0.65 | 0.537 → 0.069 | Medians of three JDK 25 JVM runs, alternating order, 20 warmups/40 measurements per mode; 64 input bitmaps × 2,048 inserted IDs. CPU/heap allocation include reader-context lifecycle and intermediate extraction. Synthetic inputs, not a JMH benchmark or customer QPS measurement. Dense input allocation improves substantially; sparse input CPU and allocation regress. Actual workload shape and end-to-end validation are required before enabling or considering a default change. ## Validation - 186 focused tests passed (zero failures/skips), including native reader V4/V5/V6 and legacy formats, existing bitmap query tests, and StarTree SV/MV queries with the option off/on. - 16 new mapped-file cases cover four format versions and four codecs, direct/read-only views, oversized/repeated/reverse reads, callback failure, malformed null rows, chunk reuse, distinct group ownership, results after reader/buffer close, and wire serialization. The group-isolation assertion was strengthened and all 16 rerun successfully. - Spotless, checkstyle and license checks passed for `pinot-spi`, `pinot-segment-spi`, `pinot-segment-local` and `pinot-core`. - No deployment or claimed customer performance gain. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
