LuciferYang opened a new pull request, #1350: URL: https://github.com/apache/paimon-rust/pull/1350
### Purpose Linked issue: close #1349 The `rbm64` aggregator decodes every Java bitset container into one range per set bit and sets the bits back one at a time when it encodes a bitset. This keeps bitset containers as their 1024 words through the union. ### Brief change log - A decoded container is either a normalized range list (run and array containers) or the 1024 words of a bitset container. - `Container::union` handles the four combinations. Two range lists are concatenated and normalized, two bitsets are ORed word by word, and a range list is set into a bitset a word at a time. - `encode` keeps the existing size rule for choosing run, array or bitset. For a bitset it counts runs from the words, and writes the words directly when the result is still a bitset. For the same set of values the output bytes are the same as before. Writing 20 rows for one key, each with 16 bitset containers that have about half of their bits set at random (release build, in-memory FileIO, 3 runs): 333 to 335 ms before, 2.8 to 6.0 ms after. I also compared the old and new decode/encode/union code on about 75k random containers and 45k random unions, with overflow checks on: same bytes, same errors. ### Tests - `decodes_java_bitset_containers_as_words`: a fixture written by RoaringBitmap 1.2.1 (`goldens/GenerateRoaring64Bitset.java`, two bitset containers and one array container) decodes to words and ranges with the expected values and survives an encode/decode round trip. - `unions_java_bitset_containers_with_ranges_and_bitsets`: bitset with array, array with bitset, and bitset with bitset (the result is all 65536 values and is written as one run), plus a bitset key that only the input has. The output bytes must equal encoding the expected values as ranges. - `word_runs_match_a_bit_by_bit_scan`: for six word patterns, including runs that cross word boundaries, the runs and run count match a bit-by-bit scan, and encoding the words gives the same bytes as encoding the ranges. - `set_ranges_matches_a_bit_loop`: setting ranges a word at a time, including start and end in one word, end at 65535, and the full container. ### API and Format None. The serialized bitmap bytes do not change. ### Documentation None. Generated-by: Claude Opus -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
