JustinBinber opened a new issue, #11075:
URL: https://github.com/apache/paimon/issues/11075

   ### Search before asking
   
   - [x] I searched existing issues. #6834 introduces BTree global indexes but 
does not address null-heavy shuffle cost.
   
   ### Motivation
   
   Sorted global index builds currently shuffle one normalized record per null 
input, even though BTree and Bitmap writers ultimately store null row IDs as 
bitmaps. Null-heavy scalar columns therefore pay unnecessary network and 
per-record shuffle overhead.
   
   For example, on a row-tracked Data Evolution append table with 200,000 rows 
and 90% null index keys, the current Spark range exchange writes about 1.1 MB 
of shuffle data. Dropping null records is not correct: their row IDs are 
required for null predicates and index coverage.
   
   ### Solution
   
   Batch null row IDs into bounded compressed bitmaps only when the observed 
null density makes aggregation worthwhile. Sparse and non-null data should 
retain the original compact wire representation rather than paying for an extra 
payload column or Java-object serialization.
   
   Range sampling must use original rows. Batches must not cross assigned 
destinations, and must be expanded before local sorting so that file rotation 
and query layout remain unchanged for identical sampled boundaries. Keep memory 
bounded, preserve incremental-build semantics, and compare real index files and 
query read bytes, not only result counts.
   
   ### Anything else?
   
   Scope: Spark and Flink single-column BTree/Bitmap global index builds. 
Composite and Multivalue builds should retain their current behavior. No 
persisted index format change or new table option is needed.
   
   I have prepared an implementation with targeted correctness, 
file-equivalence and transport-cost tests, and would like to work on this issue.
   
   ### Are you willing to submit a PR?
   
   - [x] I am willing to submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to