LuciferYang opened a new pull request, #1350:
URL: https://github.com/apache/paimon-rust/pull/1350

   ### Purpose
   
   Linked issue: close #1349
   
   The `rbm64` aggregator decodes every Java bitset container into one range 
per set bit and sets the bits back one at a time when it encodes a bitset. This 
keeps bitset containers as their 1024 words through the union.
   
   ### Brief change log
   
   - A decoded container is either a normalized range list (run and array 
containers) or the 1024 words of a bitset container.
   - `Container::union` handles the four combinations. Two range lists are 
concatenated and normalized, two bitsets are ORed word by word, and a range 
list is set into a bitset a word at a time.
   - `encode` keeps the existing size rule for choosing run, array or bitset. 
For a bitset it counts runs from the words, and writes the words directly when 
the result is still a bitset. For the same set of values the output bytes are 
the same as before.
   
   Writing 20 rows for one key, each with 16 bitset containers that have about 
half of their bits set at random (release build, in-memory FileIO, 3 runs): 333 
to 335 ms before, 2.8 to 6.0 ms after. I also compared the old and new 
decode/encode/union code on about 75k random containers and 45k random unions, 
with overflow checks on: same bytes, same errors.
   
   ### Tests
   
   - `decodes_java_bitset_containers_as_words`: a fixture written by 
RoaringBitmap 1.2.1 (`goldens/GenerateRoaring64Bitset.java`, two bitset 
containers and one array container) decodes to words and ranges with the 
expected values and survives an encode/decode round trip.
   - `unions_java_bitset_containers_with_ranges_and_bitsets`: bitset with 
array, array with bitset, and bitset with bitset (the result is all 65536 
values and is written as one run), plus a bitset key that only the input has. 
The output bytes must equal encoding the expected values as ranges.
   - `word_runs_match_a_bit_by_bit_scan`: for six word patterns, including runs 
that cross word boundaries, the runs and run count match a bit-by-bit scan, and 
encoding the words gives the same bytes as encoding the ranges.
   - `set_ranges_matches_a_bit_loop`: setting ranges a word at a time, 
including start and end in one word, end at 65535, and the full container.
   
   ### API and Format
   
   None. The serialized bitmap bytes do not change.
   
   ### Documentation
   
   None.
   
   Generated-by: Claude Opus
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to