HarshilMital opened a new pull request, #7199: URL: https://github.com/apache/datafusion-comet/pull/7199
## Which issue does this PR close? Part of #7080. ## Rationale for this change The high-partition group only ran with `CompressionCodec::None`, but a wide shuffle runs with LZ4 by default. At high partition counts every block is tiny, so the per-block codec cost (a fresh LZ4 frame encoder, its header and end mark) is spread over very few rows and was never measured. ## What changes are included in this PR? Each partition count in `shuffle_writer_high_partition` (200, 2000 and 8000 hash partitions, end to end) now also runs with `Lz4Frame`, next to the existing `None` rows. The existing IDs are unchanged. `sample_size(10)`, as the group already uses. ### Results **Machine:** 8 vCPU Arm Neoverse-V2 (aarch64), 31 GiB RAM, rustc 1.99.1 **Commit:** `174bbc6`. Median with Criterion's 95% interval. Reproduce from `native/`: ```shell cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- 'shuffle_writer_high_partition' --save-baseline main ``` | Partitions | `compression=None` | `compression=Lz4Frame` | Lz4Frame vs None | |---:|---:|---:|---:| | 200 | 14.24 ms (14.24 ms – 14.25 ms) | 6.87 ms (6.85 ms – 6.88 ms) | -52% | | 2000 | 21.03 ms (20.84 ms – 21.23 ms) | 21.74 ms (21.69 ms – 21.80 ms) | +3% | | 8000 | 44.02 ms (43.90 ms – 44.20 ms) | 46.63 ms (46.49 ms – 46.82 ms) | +6% | Both columns come from the same run. The codecs cross over as partitions grow. At 200 partitions (about 400 rows per partition) LZ4 halves the write, most likely because it sends far fewer bytes to the file. By 2000 they are nearly level (LZ4 3% slower), and at 8000 (about 10 rows per partition) LZ4 is 6% slower: the blocks are too small to compress well, so it mostly adds its per-block cost. ## How are these changes tested? This PR only adds benchmark code under `native/shuffle/benches/`; no production code changes. - `cargo fmt --all -- --check` and `cargo clippy -p datafusion-comet-shuffle --all-targets -- -D warnings` pass. - `cargo bench -p datafusion-comet-shuffle --bench shuffle_writer --no-run` builds. - Every new case runs once in Criterion's test mode: `cargo bench -p datafusion-comet-shuffle --bench shuffle_writer -- --test 'shuffle_writer_high_partition/.*compression=Lz4Frame'`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
