andygrove opened a new pull request, #2797: URL: https://github.com/apache/datafusion-ballista/pull/2797
# Which issue does this PR close? No single issue. Split out of #2792. Related: #2520 (performance review tracking). # Rationale for this change DataFusion caps spill files at 100 GB per runtime. At TPC-H SF1000 on two executors, q5 and q18 spill more than that and fail, although the executor's work dir is on a 1 TB volume. # What changes are included in this PR? When `--max-temp-directory-size` is not set, the executor uses the space available on the work dir's filesystem at startup (`statvfs`, unix only), never less than DataFusion's 100 GB default. If the free space cannot be read, DataFusion's default applies as before. A configured value is used unchanged. # Are there any user-facing changes? Yes. With `--max-temp-directory-size` unset, the executor's spill limit is the work dir's free space at startup instead of 100 GB. The flag's help text is updated. This lets spills fill a disk the executor shares with other things, so set the flag where that matters. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
