LuciferYang opened a new issue, #1349:
URL: https://github.com/apache/paimon-rust/issues/1349

   ### Search before asking
   
   - [x] I searched in the 
[issues](https://github.com/apache/paimon-rust/issues) and found nothing 
similar.
   
   ### Motivation
   
   The `rbm64` field aggregator decodes both serialized Roaring64Bitmap values, 
unions them and encodes the result for every merged row. A bitset container 
(kind 1) is decoded into one `(low, low)` range per set bit, up to 65536 
tuples. Those are sorted and merged into runs, and when the result is a bitset 
again, encoding sets the 1024 words back bit by bit. Java's 
`Roaring64Bitmap.or` ORs two bitset containers word by word.
   
   Writing 20 rows for one key, each carrying 16 bitset containers with about 
half of their bits set at random, takes about 334 ms on main (release build, 
in-memory FileIO; the write buffer runs the aggregation merge).
   
   ### Solution
   
   Keep bitset containers as 1024 words after decoding. Two bitsets are ORed 
word by word, a run or array container is set into a bitset a word at a time, 
and a result that is still a bitset is written out as is. The container kind 
choice and the encoded bytes stay the same.
   
   ### Willingness to contribute
   
   - [x] I'm willing to submit a PR!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to