adriangb opened a new pull request, #27602:
URL: https://github.com/apache/datafusion/pull/27602

   ## Which issue does this PR close?
   
   - Part of #22758
   
   ## Rationale for this change
   
   Each `RefCountedTempFile` currently holds an `Arc<NamedTempFile>`, which 
keeps an open OS file descriptor for the entire lifetime of the spill file even 
while it sits idle on disk.
   
   Under tight memory limits (e.g., 256 MB–512 MB with 16 partitions), 
operators that accumulate many spill runs—such as a hash aggregate before its 
final merge—hold one file descriptor per live spill file. On ClickBench Q18 at 
a 256 MB limit, this reaches ~1,000 live runs and fails with `Too many open 
files` (`EMFILE`) under the default `ulimit -n` of 1024.
   
   `RefCountedTempFile::read_stream` already opens the file by path on demand. 
Holding `TempPath` instead of `NamedTempFile` means live spill files hold no 
file descriptors, so descriptor usage scales with active open writers and 
readers rather than total live spill files.
   
   ## What changes are included in this PR?
   
   - Store `Arc<TempPath>` instead of `Arc<NamedTempFile>` in 
`RefCountedTempFile` (`datafusion/execution/src/disk_manager.rs`), converting 
the created file via `.into_temp_path()`.
   - Open the path on demand in `OpenOptions::new().append(true)` mode in 
`RefCountedTempFile::open_writer`, preserving shared-offset append behavior 
across multiple writers on the same spill file.
   - Remove `RefCountedTempFile::inner(&self) -> &NamedTempFile` (unused in the 
workspace).
   - Add a Unix unit test `live_spill_files_hold_no_file_descriptors` verifying 
that 400 live spill files do not accumulate open file descriptors and still 
read back and clean up on drop.
   
   ## Are these changes tested?
   
   - `cargo test -p datafusion-execution --lib` (including the new 
`live_spill_files_hold_no_file_descriptors` test)
   - `cargo clippy -p datafusion-execution --all-targets -- -D warnings`
   - `cargo fmt --all -- --check`
   
   ## Are there any user-facing changes?
   
   Breaking public API change: `RefCountedTempFile::inner(&self) -> 
&NamedTempFile` is removed because `RefCountedTempFile` no longer holds an open 
`NamedTempFile`. Callers that need the path can use 
`RefCountedTempFile::path()`, and callers that need to write or read can use 
the `SpillFile` trait methods.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to