jayzhan211 opened a new pull request, #26017:
URL: https://github.com/apache/datafusion/pull/26017
## Which issue does this PR close?
- Follow-up to #26015 (stacked on it; review only the last commit until it
merges).
## Rationale for this change
The sort-merge fallback in #26015 is triggered by a failed memory
reservation,
so it never happens under the default unbounded memory pool: the build side
just
grows. This adds an explicit per-partition cap that triggers the same
fallback
whether or not a memory limit is configured, and makes the switch
reproducible.
## What changes are included in this PR?
- New option `datafusion.execution.hash_join_max_build_size`
(`Option<usize>` bytes per output partition, default `NULL` = no limit).
Once
a partition's reserved build side grows past it, the partition falls back
to
the sort-merge join exactly as on a failed reservation.
- Documented as not needed for memory safety (a memory limit already triggers
the fallback) and as possibly temporary, to be removed once hash joins
spill
natively.
## What is the testing strategy for this PR?
- `exec.rs`: `sort_merge_fallback_on_max_build_size` falls back on the cap
alone
with no memory limit and returns the same rows as the in-memory join.
- `config_matrix` gains a "no memory limit, cap under the build side" row.
- `a_promised_probe_ordering_wins_over_the_fallback`: with the cap set, a
join
whose `ORDER BY` was pushed below it keeps the probe order and does not
fall
back.
- `information_schema.slt` and `configs.md` updated.
## Are there any user-facing changes?
One new configuration option, documented in the configuration guide.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
