peterxcli opened a new pull request, #5727: URL: https://github.com/apache/datafusion-comet/pull/5727
## Which issue does this PR close? Closes #5446. ## Rationale for this change The context lifetimes introduced by #5038 can now reuse Arrow's FlatBufferBuilder after #5262. This validates the allocation savings through the production shuffle block encoder and records a repeated same-machine timing comparison. Production writer behavior and dependencies are unchanged; the checked-in lockfile selects DataFusion 55.0.0 and Arrow 59.3.0. ## What changes are included in this PR? - Add paired fresh/reused-context Criterion cases for mixed, wide, and nested schemas, 128/8192 rows, and None/LZ4/Snappy/Zstd(1). - Extend the existing test allocation observer and add an allocation regression with byte equality and IPC round-trips. Reuse saves 9 allocation/reallocation calls and 1,064 requested bytes per block for four primitive columns, or 12 calls and 8,232 bytes for 50 columns, consistently across three repetitions and four codecs. RSS and dictionary StreamWriter paths are negative controls and show no savings. - Record all results, commands, and limitations in `native/shuffle/benches/ipc-context-reuse.md`, preserving the earlier noisy measurements separately. On the quieter Apple M4 rerun, all 128-row cases improve consistently: median reductions of 2.0–16.8%. Mixed uncompressed batches decrease from 1.644 to 1.368 µs (15.7–17.4% across three passes). Full-size compressed batches are approximately neutral (-0.2% to +0.6% median reductions); the apparent wide-LZ4 slowdown changes direction between passes (-0.7% to +0.8%), so it is not a repeatable regression. This measures block encoding, not end-to-end Spark speedup. Outer compression encoders remain per-block, and first-use context allocation is excluded from the warmed reuse arm. ## How are these changes tested? - `cargo test --locked --release -p datafusion-comet-shuffle --lib`: 119 passed, 0 failed, including the allocation regression, IPC round-trips, and writer tests. - Three complete Criterion passes (48 cases each), with 10 samples, 0.5-second warmup, and 1-second measurement time; new baselines preserve the original runs. - `cargo fmt --all --check`, Prettier on the results note, and `git diff --check`. - Full native/JVM build: `JAVA_HOME=/path/to/jdk17 make FEATURES_ARG=--release PROFILES=-Prelease` passed (Spark 4.1 / Scala 2.13). JVM tests are skipped by this package build; the 119 tests above are native. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
