siddharthteotia opened a new pull request #5548: URL: https://github.com/apache/incubator-pinot/pull/5548
## Description In PR https://github.com/apache/incubator-pinot/pull/5409, we implement efficient vectorized bit unpacking reader (no format change, just new algorithms) for dictionary encoded bit-compressed forward index. This PR starts the integration process which is divided into multiple parts - **Part 1:** Use for SV reader, MV reader for power of 2 bit encodings. Fallback to older reader for non power of 2 encodings. Since the format hasn't changed, we can leverage the new reader on existing format/segments. This PR is for Part 1 - **Part 2:** Part 2 can be divided into sub-parts: - **Part 2.1** Create a new writer that going forward uses LITTLE ENDIAN format. We need to move all our storage structures to native byte order (LE on pretty much all systems). See PR https://github.com/apache/incubator-pinot/pull/5511. This can be a good starting point. - **Part 2.2** The new writer can either choose to only have power of 2 bit encodings. So going forward for new segments, we will round up to nearest power of 2. The other alternative is to support non power of 2 upto 16 and from there on round off to 32. Either way, there is no impact on existing segments -- due to fallback implemented in Part 1 (this PR) ## Upgrade Notes Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion) No Does this PR fix a zero-downtime upgrade introduced earlier? No Does this PR otherwise need attention when creating release notes? Things to consider: No ## Release Notes None ## Documentation If you have introduced a new feature or configuration, please add it to the documentation as well. See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
