kamilk-pal opened a new pull request, #27562: URL: https://github.com/apache/datafusion/pull/27562
## Rationale for this change The memory-limited (spilling) fallback of `NestedLoopJoinExec`, added in 54.0.0 by #21448, can panic or silently return incorrect results on `branch-54`. Fixing it on 54.x would take at least three bug fixes: - #24677: the build side is executed twice, which panics on a `RepartitionExec` and loses rows from file scans (#24676) - #24675: LEFT-type joins with a multi-partition probe side emit wrong unmatched rows (#22641) - #24746: unmatched probe-side rows are dropped after the last chunk (#24745) These sit on top of continued rework of the feature on `main`: #21636, #21833, #22038, #25004, #25533, #25542. More reports keep arriving, e.g. #25701 (an empty result instead of spilling, reported on 55.1.0). All of it is first released in 56.0.0. Rather than backporting that series, this PR disables the fallback on `branch-54`. A nested loop join whose build side does not fit in memory fails with `ResourcesExhausted`, as it did before 54.0.0. ## What changes are included in this PR? - `ENABLE_MEMORY_LIMITED_EXECUTION = false` in `nested_loop_join.rs`, checked where `execute` decides whether it may fall back. The fallback code itself is unchanged. - Tests that need the fallback are `#[ignore]`d, a new test checks that every join type fails with `ResourcesExhausted`, and `nested_loop_join_spill.slt` expects the error. ## Are there any user-facing changes? Yes. On 54.x, a nested loop join whose build side exceeds the memory limit fails with `ResourcesExhausted` instead of spilling. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
