kosiew opened a new pull request, #22671:
URL: https://github.com/apache/datafusion/pull/22671
## Which issue does this PR close?
* Closes #22641.
## Rationale for this change
The memory-limited NestedLoopJoin fallback path can produce incorrect
results for joins that require final left-side emission when the right side has
multiple partitions.
The fallback implementation tracks match state locally for each probe
partition. Without global coordination across all right partitions, a partition
may incorrectly emit unmatched, semi, anti, or mark results for left rows that
actually match in another partition. This can lead to incorrect query results
under memory pressure.
To preserve correctness, this change disables the memory-limited fallback
for join types that depend on final left-side emission when the right side is
partitioned. In these cases, the operator will continue to fail with resource
exhaustion rather than produce incorrect output.
## What changes are included in this PR?
* Generalize the existing FULL JOIN fallback guard to cover all joins that
require final left-side emission when `right_partition_count > 1`.
* Replace the previous `full_join_multi_partition` check with a
`left_final_multi_partition` check based on
`need_produce_result_in_final(join_type)`.
* Update comments to document why memory-limited fallback is unsafe for
these join types without cross-partition left-match tracking.
* Adjust memory-limited join tests so that:
* Supported join types continue to verify successful spill-based fallback.
* `LEFT`, `LEFT SEMI`, `LEFT ANTI`, `LEFT MARK`, and `FULL` joins verify
that fallback remains disabled and returns a resource exhaustion error under
tight memory limits.
* Add a dedicated regression test:
* `test_nlj_memory_limited_left_join_multi_partition_fallback_disabled`
* Verifies that a multi-partition LEFT JOIN does not use the fallback path
under memory pressure.
* Verifies that a single-partition LEFT JOIN can still spill and complete
successfully.
## Are these changes tested?
Yes.
The following tests were added or updated:
* Added:
* `test_nlj_memory_limited_left_join_multi_partition_fallback_disabled`
* `assert_multi_partition_join_oom`
* Updated memory-limited fallback coverage to verify:
* Successful spill-based fallback for supported join types (`Inner`,
`Right`, `RightSemi`, `RightAnti`, `RightMark`).
* Resource exhaustion behavior for left-emitting join types with
multi-partition right inputs (`Left`, `LeftSemi`, `LeftAnti`, `LeftMark`,
`Full`).
## Are there any user-facing changes?
Yes.
Under memory pressure, NestedLoopJoin will no longer attempt the
memory-limited fallback path for joins that require final left-side emission
when the right side has multiple partitions. Instead of potentially returning
incorrect results, these queries will fail with a resource exhaustion error
until cross-partition coordination is implemented.
This change prioritizes correctness and ensures join results do not vary
based on memory availability.
## LLM-generated code disclosure
This PR includes LLM-generated code and comments. All LLM-generated content
has been manually reviewed and tested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
