jianguotian opened a new pull request, #91: URL: https://github.com/apache/paimon-mosaic/pull/91
## Purpose Reduce dictionary-tracking and nullable-write overhead in `bucket_writer.rs` while preserving existing encodings, value ordering, and file-format compatibility. ## Changes - defer dictionary allocation and hashing while a column remains CONST - activate dictionary tracking when the first different value appears - stop per-value dictionary tracking after dictionary encoding is disabled - copy Arrow null bitmaps in chunks and fast-path all-null batches - write nullable timestamp-millisecond values by iterating valid indices only - add focused coverage for dictionary state transitions, byte and entry limits, bitmap offsets, sliced arrays, and append offsets ## Performance evidence A nullable timestamp benchmark decreased from 27,776 us to 19,532 us, approximately 29.7%, for that scenario. This is a combined-path result rather than a whole-process claim. CONST-heavy columns also avoid unnecessary dictionary allocation and hashing, but no separate end-to-end gain is claimed for that path. ## Compatibility - no Mosaic encoding or file-format change - dictionary insertion order is preserved - no FFI or public API change ## Tests - `cargo fmt --all -- --check` - `cargo clippy --locked --offline --all-targets --workspace -- -D warnings` - `cargo test --locked --offline --workspace` - `cargo build --locked --offline --all-targets` - `cargo deny check licenses` - `python3 tools/dependencies.py check` - `git diff --check HEAD^` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
