JustinBinber opened a new issue, #11075: URL: https://github.com/apache/paimon/issues/11075
### Search before asking - [x] I searched existing issues. #6834 introduces BTree global indexes but does not address null-heavy shuffle cost. ### Motivation Sorted global index builds currently shuffle one normalized record per null input, even though BTree and Bitmap writers ultimately store null row IDs as bitmaps. Null-heavy scalar columns therefore pay unnecessary network and per-record shuffle overhead. For example, on a row-tracked Data Evolution append table with 200,000 rows and 90% null index keys, the current Spark range exchange writes about 1.1 MB of shuffle data. Dropping null records is not correct: their row IDs are required for null predicates and index coverage. ### Solution Batch null row IDs into bounded compressed bitmaps only when the observed null density makes aggregation worthwhile. Sparse and non-null data should retain the original compact wire representation rather than paying for an extra payload column or Java-object serialization. Range sampling must use original rows. Batches must not cross assigned destinations, and must be expanded before local sorting so that file rotation and query layout remain unchanged for identical sampled boundaries. Keep memory bounded, preserve incremental-build semantics, and compare real index files and query read bytes, not only result counts. ### Anything else? Scope: Spark and Flink single-column BTree/Bitmap global index builds. Composite and Multivalue builds should retain their current behavior. No persisted index format change or new table option is needed. I have prepared an implementation with targeted correctness, file-equivalence and transport-cost tests, and would like to work on this issue. ### Are you willing to submit a PR? - [x] I am willing to submit a PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
