sdf-jkl opened a new pull request, #11061: URL: https://github.com/apache/arrow-rs/pull/11061
# Which issue does this PR close? None. This is an experimental follow-up stacked on #10977, based on its head `f6e19d07a3a574b4c83ff32559f2f91332fc6281`. It includes that PR's commits and should remain a draft while the underlying PFOR proposal is under review. The incremental change is [f0ece7446](https://github.com/sdf-jkl/arrow-rs/commit/f0ece7446b527ab255c25be4cf034c2acb08a606). # Rationale for this change The PFOR implementation in #10977 unpacks residuals sequentially and then adds the frame of reference in a separate pass. Its default 1,024-value vectors also fit FastLanes packing. This experiment adds an opt-in FastLanes layout with frame addition fused into unpacking, and compares both implementations with the same frame selection, patches, and delta logic. # What changes are included in this PR? - Byte-oriented FastLanes kernels for 32-bit and 64-bit values, adapted from the earlier local ALP experiment. They read and write unaligned little-endian words directly, accommodating variable byte offsets in PFOR payloads. The upstream `fastlanes` crate is used only as a dev dependency for an independent test oracle. - Experimental packing mode 1, selected with `with_fastlanes_enabled(true)`. Each vector uses complete 1,024-value FastLanes blocks followed by a sequentially packed tail, with no padding or encoded-size increase. Mode 0 remains the default, and the decoder handles both modes. - Fused wrapping frame addition during FastLanes unpacking. Delta reconstruction remains the existing single prefix sum; this does not introduce UTL delta chains. - FastLanes arms in the PFOR encode and decode benchmarks, including delta-enabled variants, explicit output consumption, and equal-encoded-size assertions. # Are these changes tested? Passed: ```text cargo test -p parquet --lib --offline encodings:: # 197 tests cargo test -p parquet --lib --offline arrow_writer_pfor # 2 tests cargo clippy -p parquet --features experimental --lib --tests --offline -- -D warnings cargo fmt --all --check git diff --check ``` Kernel tests compare every bit width with the upstream FastLanes oracle and cover byte alignments 0–7, wrapping frame addition, and full-width values. PFOR tests exercise both packing modes, streaming decode, tails, skipping, and supported vector sizes. ## Benchmarks Recorded on one pinned logical CPU (CPU 2) of an AMD Ryzen AI 9 HX PRO 470, using Rust 1.97.1 / LLVM 22.1.6 and `-C target-cpu=native`. The benchmark uses 33 synthetic i32 columns and four synthetic i64 columns, each containing 102,400 values. The full run has 148 baseline/FastLanes pairs, with 30 samples, 0.1 s warmup, and 0.5 s measurement. Encoded sizes are identical in every pair. Geometric means across the corresponding workloads; GB/s means decimal uncompressed bytes divided by elapsed time: | Type / delta setting | Encode speedup | Baseline decode GB/s | FastLanes decode GB/s | Decode speedup | |---|---:|---:|---:|---:| | i32 / disabled | 1.13× | 20.28 | 41.54 | 2.05× | | i32 / enabled | 1.08× | 18.27 | 35.46 | 1.94× | | i64 / disabled | 1.15× | 23.56 | 40.76 | 1.73× | | i64 / enabled | 1.09× | 20.89 | 34.06 | 1.63× | These are repeated hot-cache codec measurements and include the benefit of fused frame addition. They do not measure full Parquet scan throughput. An additional standalone wall-clock sanity probe linked the same compiled library, decoded pseudorandom 21-bit i32 residuals with a nonzero frame and no delta, consumed the output, and checked all decoded values. Two rounds reversed the implementation order: | Probe | Baseline decode GB/s | FastLanes decode GB/s | |---|---:|---:| | One hot page, ~0.68 MB encoded-input/output working set | 9.05–9.21 | 36.51–37.33 | | 256 distinct pages, ~174 MB encoded-input/output working set | 7.37–7.57 | 10.85–11.10 | The larger working set reduces the advantage to roughly 1.47× in this synthetic case. This probe also excludes file I/O and full Parquet materialization. To reproduce the primary benchmark (with dependencies available locally, `--offline` may be added): ```sh RUSTFLAGS='-C target-cpu=native' taskset -c 2 cargo bench \ -p parquet --features experimental --bench pfor -- \ 'int(32|64)/(encode|decode)/PFOR' \ --sample-size 30 --warm-up-time 0.1 --measurement-time 0.5 --noplot ``` # Are there any user-facing changes? The experimental PFOR encoder gains an opt-in FastLanes packing mode. Existing default encoding behavior is unchanged. Payloads written with mode 1 require this decoder extension and cannot be read by the mode-0-only decoder in #10977. This is an experimental wire-format extension, not a proposal to change existing standard Parquet encodings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
