sundapeng opened a new pull request, #1080: URL: https://github.com/apache/paimon-rust/pull/1080
### Purpose Follow-up to #1069. When a shredded VARIANT column is read only for field extraction, every requested object field brings two leaves: its `typed_value` and its residual `value`. When every value of a field was shredded, the residual leaf is null in every row, yet it is still fetched, decompressed and decoded. On wide shredded schemas this doubles the column chunks per row group. parquet-rs also builds a codec for every column chunk, creating ZSTD compression and decompression contexts each time. With 200 fields shredded as DOUBLE out of a ~5,500-key object (about 400 leaves), this per-chunk overhead was the main CPU cost left after #1069. In the profile, creating the ZSTD contexts alone took about 0.1 core-second per batch. ### Brief change log - `null_shredded_residuals` collects object-field residual leaves (`typed_value.<key>.value`) whose statistics report as many nulls as values in every row group of the file. Leaves without statistics are kept. - `read_type_projection_mask` leaves those leaves out of the projection for Variant extraction reads. Full Variant reads and columns referenced by predicates are unchanged. - Both extraction paths already treat a missing residual as "no residual value", so results are unchanged. ### Tests New test: - `shredded_variant_extraction_skips_null_residuals`: a shredded Parquet file in which one field is fully shredded and another has a residual value. The all-null residual is not read, the other is, and the projection equals the in-memory assembly. Commands: - `cargo test -p paimon --lib`: 3,841 passed, 6 ignored. - `cargo test -p paimon-datafusion --test variant_pushdown`: 11 passed. - `cargo clippy --locked --all-targets --workspace --features fulltext,vortex -- -D warnings` and `cargo fmt --all -- --check`: passed. Benchmark: PyTorch DataLoader on a 60-vCPU host (8 ranks x 6 workers, batch 16) reading the wide shredded table above. Throughput is the median of the runs. | Build | Runs | samples/s | CPU per batch | Peak memory | |---|---:|---:|---:|---:| | #1069 | 16 | 499 | 1.73 core-s | 125 GiB | | #1069 + this change | 3 | 524 | 1.59 core-s | 98 GiB | - In the profile, pypaimon-rust CPU per batch fell from 0.70 to 0.535 core-seconds, the same as the unshredded table. Creating ZSTD compression and decompression contexts took half the time it did before. - The full output hash of every batch is unchanged. ### API and Format No new public API, options, dependencies or storage format changes. ### Documentation None. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
