pingzh opened a new pull request, #5722: URL: https://github.com/apache/datafusion-comet/pull/5722
## Which issue does this PR close? Closes #5535. ## Rationale for this change The JVM remote shuffle reader copies and parses the same expected schema on every block. Retain the parsed Spark types for the iterator's lifetime, and add a reproducible benchmark for the remaining remote decode cost. ## What changes are included in this PR? - Lazily create a native decoder on the first remote block and reuse it across subsequent blocks. Release it once on iterator close, including early completion and cleanup failures. Empty and local streams allocate no decoder. - Keep Arrow buffer/offset and logical type validation mandatory. Decoder initialization failures remain outside remote corruption reporting. - Add 16 Criterion cases covering plain and dictionary batches across None, LZ4, Snappy, and Zstd(1), with fixture correctness checks and documented results. ### Remote decode measurements Linux x86_64, AMD EPYC 9V74, Rust 1.97.1, DataFusion 55.0.0, Arrow 59.3.0; default release profile, 100 samples, 3-second warm-up, 5-second target measurement time. Each block contains 8192 rows and 20 columns (10 Int32, 10 Utf8). Times are Criterion's estimated mean in microseconds per block. | Encoding | Codec | Frame bytes | Local | Remote | Remote/local | | --- | --- | ---: | ---: | ---: | ---: | | plain | None | 2,075,868 | 106.7 | 290.5 | 2.72x | | plain | LZ4 | 704,026 | 410.4 | 602.9 | 1.47x | | plain | Snappy | 785,894 | 577.1 | 774.1 | 1.34x | | plain | Zstd(1) | 248,249 | 1577.5 | 1776.2 | 1.13x | | dictionary | None | 709,660 | 67.0 | 751.5 | 11.21x | | dictionary | LZ4 | 342,473 | 191.3 | 877.8 | 4.59x | | dictionary | Snappy | 370,097 | 259.6 | 943.4 | 3.63x | | dictionary | Zstd(1) | 205,611 | 359.3 | 1053.5 | 2.93x | The remote dictionary path also expands values for the JVM importer, so that gap includes normalization. Measurements include decompression, IPC decoding, validation/normalization, and batch destruction; they exclude transport, JNI, Arrow FFI, and schema initialization. They do not measure the schema-cache speedup or establish an end-to-end advantage over Celeborn's row decoder. Fixtures differ from the issue's original measurements. Validation remains required despite its cost. ## How are these changes tested? - `CometCelebornShuffleReaderSuite` on Spark 4.1: 64 passed against the new release JNI library, including multi-frame reuse, later-frame schema mismatch, initialization failure classification, and cleanup lifecycle coverage. - `cargo test --release -p datafusion-comet-shuffle --lib`: 118 passed. - `shuffle_reader`: all 16 correctness cases and the full Criterion measurement run passed. - Native release build, full JVM reactor test compilation, Spotless, Scalastyle, `cargo fmt --check`, and `git diff --check` passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
