peterxcli opened a new pull request, #5805: URL: https://github.com/apache/datafusion-comet/pull/5805
## Which issue does this PR close? Part of #5792. This adds the measurement that issue asks for; it does not change the decode path. ## Rationale for this change Every shuffle block is a self-contained Arrow IPC stream, so `read_single_batch` builds a fresh `StreamReader` per block and parses the schema flatbuffer once per block, even though every block in a shuffle carries the same schema and the reducer already knows it from the plan protobuf. The write side already avoids the mirror image of this: `ShuffleBlockWriter` encodes the schema once in `try_new` and writes the pre-encoded bytes verbatim (`SchemaEncoding::Precoded`). #5198 recorded that this read-side item had no benchmark coverage, and #5792 was filed as unmeasured. Removing the per-block parse means replacing `StreamReader` with `RecordBatchDecoder` and hand-rolling IPC message framing, which has to reproduce the existing guards for truncated LZ4 frames, trailing data, more than one batch per frame, and the `skip_validation` split between trusted local and untrusted remote blocks. That is worth doing against a number rather than a hunch, so this PR establishes the number first. ## What changes are included in this PR? A new `native/shuffle/benches/shuffle_reader.rs`, parameterized by column count and rows per block, measuring the schema parse separately from the full block decode. No production code changes. Results on an M-series laptop: | shape | decode | schema parse | share | | --- | --- | --- | --- | | 5 col x 64 row | 1.93 us | 1.14 us | 59% | | 5 col x 512 row | 2.38 us | 0.91 us | 38% | | 5 col x 8192 row | 10.99 us | 0.86 us | 8% | | 50 col x 64 row | 12.77 us | 6.03 us | 47% | | 50 col x 512 row | 17.89 us | 6.05 us | 34% | | 50 col x 8192 row | 218 us | 6.05 us | 3% | The parse cost is constant per block and independent of row count, so its share is set by how many rows land in a block. That is largest exactly where #5792 predicted: wide shuffles, where rows per partition are few, and repeated spilling, where each spill round emits its own block per partition. At the 8192 row blocks a default 200 partition shuffle produces it is 3 to 8 percent, so the optimization is worth doing but its payoff is conditional on shuffle width rather than general. The `50 col x 8192 row` decode row was noisy under a shortened measurement window, [161 us, 334 us]. The others were tight, and that row's conclusion does not depend on the precision. ## How are these changes tested? Benchmark-only change, so there is no behaviour to test. Run with: ``` cargo bench --bench shuffle_reader -p datafusion-comet-shuffle ``` The benchmark encodes its blocks through `ShuffleBlockWriter` and decodes them through `read_ipc_compressed`, the same paths the shuffle write and read sides use, so it fails if either changes incompatibly. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
