andygrove opened a new issue, #3834: URL: https://github.com/apache/datafusion-comet/issues/3834
## Description In `MultiPartitionShuffleRepartitioner::shuffle_write()`, spill files are copied to the final shuffle output using `BufReader`: ```rust let mut spill_file = BufReader::new(File::open(spill_path)?); std::io::copy(&mut spill_file, &mut output_data)?; ``` The `BufReader` wrapper is counterproductive here because: 1. `std::io::copy` already uses an internal buffer for the copy 2. On Linux, `std::io::copy` with raw `File`-to-`File` can use `copy_file_range` / `sendfile` for kernel zero-copy, but wrapping in `BufReader` defeats this specialization since the source is no longer a `File` ## Proposed Change Remove the `BufReader` wrapper and pass the raw `File` handle directly: ```rust let mut spill_file = File::open(spill_path)?; std::io::copy(&mut spill_file, &mut output_data)?; ``` This is a one-line change in `native/shuffle/src/partitioners/multi_partition.rs` (in the `shuffle_write` method). Note: the same optimization was already applied to the new `ImmediateShufflePartitioner`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
