Sunninsky opened a new pull request, #27354: URL: https://github.com/apache/datafusion/pull/27354
## Which issue does this PR close? - Part of #26868 (the 55.3.0 patch release tracking issue). - Backport of #26027 to `branch-55`. #26027 fixed #26001. ## Rationale for this change With a sort-merge join (`datafusion.optimizer.prefer_hash_join = false`), a `LEFT`, `RIGHT` or `FULL` join with an extra join filter fails with `Arrow error: Invalid argument error: Column '...' is declared as non-nullable but contains null values` when the NULL-padded side has `NOT NULL` columns referenced by the filter. Hash join returns the correct result for the same queries. This is a failed query, which makes it eligible for backport as a correctness fix. When a streamed row has no match but the buffered side still holds a batch, the NULL-padded row is materialized in the same chunk as matched rows, so its NULL buffered-side columns reach filter evaluation, whose intermediate schema declares them non-nullable. #26027 fixed it on `main` (labelled `v56.0.0`); it is on `branch-56` but not on `branch-55`, so it is in neither 55.1.0 nor 55.2.0. It was requested for 55.3.0 on #26868 (https://github.com/apache/datafusion/issues/26868#issuecomment-6791317393). ## What changes are included in this PR? - A `git cherry-pick -x` of 8248a579695b92b7ca94709eedd63a71689513ce (#26027, by @CuteChuanChuan). - `datafusion/physical-plan/src/joins/sort_merge_join/materializing_stream.rs`: `MaterializingSortMergeJoinStream::try_new` builds the filter schema once with the buffered-side fields marked nullable, and filter evaluation uses it. It is only built when the join needs deferred filtering (outer join with a filter); inner joins keep the original schema. The filter result of a NULL-padded row does not affect the output, so this does not change results. - `datafusion/sqllogictest/test_files/sort_merge_join.slt`: the `LEFT`, `FULL` and `RIGHT` cases from the issue, and a `LEFT` case whose filter divides by a buffered-side column. - One conflict, in `materializing_stream.rs`: on this branch `needs_deferred_filtering` takes `&Option<JoinFilter>` (on `main` it takes `Option<&JoinFilter>`), so the new local passes `&filter`. The rest is as upstream. ## Are these changes tested? Yes. On this branch (Rust 1.95.0) I ran: - `cargo test -p datafusion-sqllogictest --test sqllogictests -- join`: all 16 matching files pass, including `sort_merge_join.slt` with the new cases. - With the change in `materializing_stream.rs` reverted and the new slt cases kept, the four new queries fail with the Arrow error above (`sort_merge_join.slt` lines 990, 997, 1004 and 1013). - `cargo test -p datafusion-physical-plan --lib`: 1700 passed. - `cargo fmt --all -- --check` and `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` pass. ## Are there any user-facing changes? No. Bug fix only; no public API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
