richardstartin opened a new pull request #6320: URL: https://github.com/apache/incubator-pinot/pull/6320
## Description This PR uses some more efficient APIs from RoaringBitmap. - `RoaringBitmapWriter` is optimised for writing row ids into the bitmap sequentially. This class has a few options to control memory usage, but I've used its defaults. - A `RoaringBitmap` can be serialized to a `ByteBuffer`, which is generally a lot faster than a `DataOutput`. However, this requires mapping a file outside of the control of `PinotByteBuffer`. The buffer also has to be big endian for backward compatibility with `DataOutput`. I haven't run any benchmarks on this and am judging this on having used (and created) the same APIs for a similar problem. I tried to run the pinot-perf benchmarks but ran into errors with missing files. I would appreciate some help validating the benefits this change brings. ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) * [No] Does this PR fix a zero-downtime upgrade introduced earlier? * [ No] Does this PR otherwise need attention when creating release notes? * [ No] ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
