shinzoxD opened a new pull request, #24407: URL: https://github.com/apache/datafusion/pull/24407
## Which issue does this PR close? - Closes #24230. ## Rationale for this change `DiskManager` charges `used_disk_space` before writing a spill file so concurrent writers cannot collectively exceed `max_temp_directory_size`. On a quota miss, `FileSpillWriter::write` already rolls that reservation back. The OS `write_all` error path did not. The global counter stayed charged, `current_file_disk_usage` was left at the last successful size, and `RefCountedTempFile::Drop` subtracted too little. After one I/O error the leaked bytes stay until process exit, so later spills that fit on disk are rejected. #21882 already fixed the quota path on `main`. This is the remaining `write_all` error path called out in #24230. ## What changes are included in this PR? - If the OS write fails after the reservation, subtract the reserved bytes from `used_disk_space` before returning the I/O error. - Add a regression test that writes 400 bytes into a 1000-byte cap, then forces two 600-byte writes through a read-only handle. Without the rollback the first leak fills the cap (so the second attempt is a quota error, and a later 500-byte spill is rejected). With the fix the counter stays at 400, the 500-byte write succeeds, and dropping the file returns the counter to 0. ## Are these changes tested? Yes. - New `test_write_io_error_rolls_back_disk_usage` in `datafusion/execution/src/disk_manager.rs`. - Existing `test_rollback_on_limit_exceeded_then_drop_returns_to_zero` still covers the quota path. Ran: - `cargo test -p datafusion-execution --lib disk_manager` - `cargo clippy -p datafusion-execution --lib --all-targets -- -D warnings` ## Are there any user-facing changes? A failed spill write no longer permanently reduces remaining spill capacity. There is no API change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
