Jackie-Jiang opened a new pull request, #19818: URL: https://github.com/apache/pinot/pull/19818
## Summary `MutableRoaringBitmap.add(long, long)` regrows an array container to its exact new size on every call (`MappeableArrayContainer.iadd` allocates `CharBuffer.allocate(newcardinality)`), while the heap `ArrayContainer.iadd` grows it geometrically. Adding many short ranges to the same 65,536-id container is therefore quadratic: 4,096 one-document ranges reallocate the container 4,096 times, about 16 MB of `CharBuffer`s for an 8 KB result. #19799 addresses this for star-tree leaf ranges. This PR covers every place that adds ranges to a `MutableRoaringBitmap` in a loop, and keeps the common single-range and aggregate-only paths free of any extra copy: - `RoaringBitmapUtils.fromInclusiveRanges(List<IntPair>)` builds a bitmap from inclusive ranges in any order. No range or a single range is built directly as a `MutableRoaringBitmap`; multiple ranges are collected in a heap `RoaringBitmap` and converted once. - The sorted-index paths use it: `BlockDocIdSet.applyAnd`, `AndDocIdSet`, `OrDocIdSet` and `SortedIndexBasedFilterOperator.getBitmaps`, which now shares its adjacent-range merge with `getTrues`. A sorted-column `IN` on sparse values that each match only a few documents produces the same pattern as short star-tree leaves. - `StarTreeFilterOperator.traverseStarTree` keeps aggregated document ids in a `MutableRoaringBitmap` as before, and collects leaf ranges in a separate `RoaringBitmap` created on the first leaf. Without a leaf range the aggregated bitmap is returned as is; otherwise the leaf bitmap is converted once and the aggregated ids are OR'ed into it. Unlike #19799, the result does not depend on the order in which the traversal visits aggregated ids and leaf ranges, since the two bitmaps are merged rather than one replacing the other. The root cause is in RoaringBitmap: `MappeableArrayContainer.iadd` should grow like `ArrayContainer.iadd`. Once that is fixed upstream, `fromInclusiveRanges` can add the ranges to a `MutableRoaringBitmap` directly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
