buraksenn opened a new pull request, #24056:
URL: https://github.com/apache/datafusion/pull/24056
## Which issue does this PR close?
- No issue. Found while adding test coverage in scope of #13431.
## Rationale for this change
Filtered semi, anti, and mark sort-merge joins kept only one finished spill
file per inner key group. If a group spilled more than once, each spill
replaced the previous file, so read-back could lose earlier rows and produce
incorrect results.
## What changes are included in this PR?
Keep one `InProgressSpillFile` open per inner key group, append every
overflow to it, and finalize it before filter evaluation.
## Are these changes tested?
Yes:
```
cargo test -p datafusion-physical-plan bitwise_multi_spill_inner_key_group
cargo test -p datafusion-physical-plan
cargo test --test sqllogictests -- sort_merge_join_spill
```
## Are there any user-facing changes?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
