andygrove opened a new issue, #2690: URL: https://github.com/apache/datafusion-ballista/issues/2690
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Part of #2652. In pipelined mode, every task of a job runs at the same time, so every hash join build side in the job is resident at once. A hash join build cannot spill, and a failed pipelined task fails the whole job. #2685 adds `PipelinedJoinSelection`, which turns a `Partitioned` hash join into a sort-merge join when its estimated build size per partition is over `ballista.optimizer.hash_join_max_build_partition_bytes`. That check is per join and per partition. A job with many joins that are each just under the limit keeps all of them as hash joins, and together they can need far more memory than the limit suggests. The same applies to the per-executor total, since one executor may run several partitions of several joins at once. (Raised in review: https://github.com/apache/datafusion-ballista/pull/2685#discussion_r4407262444) **Describe the solution you'd like** A budget for the job as a whole, or per executor, that join selection checks when it plans a pipelined job. For example, add up the estimated build sizes of the hash joins that will be resident at the same time, and demote the largest ones to sort-merge joins until the total fits. The budget could come from a new config key or from the executor memory pool size. **Describe alternatives you've considered** - Keep the per-join limit and document that users should lower it for join-heavy queries. Simple, but users need to know their query shapes. - Count the job's hash tables against the executor memory pool when the job is admitted, and fall back to blocking execution if the job doesn't fit. **Additional context** Build-size estimates come from statistics and can be missing or inexact, so whatever we do needs a rule for unknown sizes. #2685 treats an unknown size as too large. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
