jianguotian opened a new pull request, #75: URL: https://github.com/apache/paimon-mosaic/pull/75
## Summary A fixed-width `CONST` column stores one value for all non-null rows. When such a column also contained nulls, the reader first decoded a compact value buffer and then copied those values into their final row positions. This PR builds the final Arrow value buffer directly, avoiding the extra allocation and copy. ## Changes - Build row-aligned Arrow values directly for fixed-width `CONST` columns with nulls. - Below 1/8 non-null density, write only valid positions. - At ordinary densities, bulk-fill the whole buffer only when every physical OS page covered by the actual output allocation contains a valid row; otherwise bulk-fill only contiguous non-null row runs. - Never use logical chunk boundaries as write ranges, because `Vec` allocations are not guaranteed to align with physical pages. - Cover nullable `INT16(0)` through the generic `Int16` path; there is no `INT16`- or zero-specific implementation. - Preserve zero values, floating-point negative zero, Boolean values, timestamps, and trailing bitmap semantics. - Ignore hidden payloads at null positions when converting nullable nanosecond timestamps, preventing false overflow errors. - Keep variable-width `Utf8`, `Binary`, and large-decimal constants on the existing compact path. - No file-format or public API changes. ## Performance ### Wide-table reader benchmark An earlier optimization-head A/B measurement, before the final RSS guard, used 12 real-world wide-table Mosaic files: | Test input | Value | |---|---:| | Files | 12 | | Columns per file | 8,691 | | Total rows | 115,631 | | Runs | 7 per file, alternating base/head order | | Metric across 12 files | Base | Optimized | Change | |---|---:|---:|---:| | Read all columns | 1,636.63 ms | 989.78 ms | **-39.5%** | | Open reader + read all columns | 1,702.51 ms | 1,055.68 ms | **-38.0%** | - All 12 files improved; per-file column-read reduction was **35.6% to 42.3%**. - Base and optimized runs produced identical row counts, column counts, and null counts. ### Page-amplification regression A 3-run exact-loop Rust probe on an aarch64/glibc host used 32M `Int64` rows at exactly 1/8 non-null density, with alternating touched/null-only logical chunks: | Materialization path | Time | Peak RSS | |---|---:|---:| | Original compact + scatter | 62-63 ms | 169,520-169,528 KiB | | Page-amplified touched chunks | 56-58 ms | 267,728-267,736 KiB | | Final non-null row runs | **33-34 ms** | **136,792 KiB** | These are local Core/materialization measurements, not end-to-end capacity or CU results. ## Validation - `cargo fmt --all -- --check` - `cargo clippy --all-targets --workspace -- -D warnings` - `cargo deny check licenses` - `python3 tools/validate_asf_yaml.py` - `cargo +1.86 check -p paimon-mosaic-core --lib` - Workspace tests: **397 passed**; the two readback cases requiring externally generated Java/Python files were skipped locally - Nullable nanosecond timestamp allocator-layout regression probe: **20,000 reads passed** - Remote CI status is tracked by the current PR checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
