andygrove opened a new issue, #2820: URL: https://github.com/apache/datafusion-ballista/issues/2820
**Is your feature request related to a problem or challenge?** When `--max-temp-directory-size` is unset, the executor's spill limit is the work dir's free space measured once at startup, never less than DataFusion's 100 GB default (#2797). Free space changes while the executor runs: other processes or shuffle files can use it up, or space can be freed after startup. A limit that tracked the filesystem would match what operators expect better. **Describe the solution you'd like** Refresh the spill limit while the executor runs. `DiskManager::set_max_temp_directory_size` takes `&self`, so a live runtime's limit can change without rebuilding it. Things to work out: - The executor would need to track every live session runtime's `DiskManager` (the shared runtime cache plus any per-task runtimes) and refresh each one on a timer. - The limit should be "spill bytes already used + space currently free", not "space currently free". Otherwise a query's own spills shrink its limit as they grow and it fails early. - How often to refresh, and whether a configured `--max-temp-directory-size` should still win (probably yes). **Describe alternatives you've considered** Keep the startup snapshot (current behavior after #2797). Set `--max-temp-directory-size` explicitly where it matters. **Additional context** Raised in review of #2797. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
