adriangb opened a new pull request, #27602: URL: https://github.com/apache/datafusion/pull/27602
## Which issue does this PR close? - Part of #22758 ## Rationale for this change Each `RefCountedTempFile` currently holds an `Arc<NamedTempFile>`, which keeps an open OS file descriptor for the entire lifetime of the spill file even while it sits idle on disk. Under tight memory limits (e.g., 256 MB–512 MB with 16 partitions), operators that accumulate many spill runs—such as a hash aggregate before its final merge—hold one file descriptor per live spill file. On ClickBench Q18 at a 256 MB limit, this reaches ~1,000 live runs and fails with `Too many open files` (`EMFILE`) under the default `ulimit -n` of 1024. `RefCountedTempFile::read_stream` already opens the file by path on demand. Holding `TempPath` instead of `NamedTempFile` means live spill files hold no file descriptors, so descriptor usage scales with active open writers and readers rather than total live spill files. ## What changes are included in this PR? - Store `Arc<TempPath>` instead of `Arc<NamedTempFile>` in `RefCountedTempFile` (`datafusion/execution/src/disk_manager.rs`), converting the created file via `.into_temp_path()`. - Open the path on demand in `OpenOptions::new().append(true)` mode in `RefCountedTempFile::open_writer`, preserving shared-offset append behavior across multiple writers on the same spill file. - Remove `RefCountedTempFile::inner(&self) -> &NamedTempFile` (unused in the workspace). - Add a Unix unit test `live_spill_files_hold_no_file_descriptors` verifying that 400 live spill files do not accumulate open file descriptors and still read back and clean up on drop. ## Are these changes tested? - `cargo test -p datafusion-execution --lib` (including the new `live_spill_files_hold_no_file_descriptors` test) - `cargo clippy -p datafusion-execution --all-targets -- -D warnings` - `cargo fmt --all -- --check` ## Are there any user-facing changes? Breaking public API change: `RefCountedTempFile::inner(&self) -> &NamedTempFile` is removed because `RefCountedTempFile` no longer holds an open `NamedTempFile`. Callers that need the path can use `RefCountedTempFile::path()`, and callers that need to write or read can use the `SpillFile` trait methods. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
