Sunninsky opened a new pull request, #27353: URL: https://github.com/apache/datafusion/pull/27353
## Which issue does this PR close? - Part of #26868 (the 55.3.0 patch release tracking issue). - Backport of #25134 to `branch-55`. #25134 fixed #25133. ## Rationale for this change On 55.2.0, a mark, semi or anti sort-merge join on a floating-point key fails when a key group ends at the end of an input batch: a Float32 or Float64 key that is NaN panics with ``called `Option::unwrap()` on a `None` value``, and a Float16 key returns `Unsupported data type in sort merge join comparator: Float16`. The main merge scan handles the same keys. These are failed queries, which makes this eligible for backport as a correctness fix. #25134 fixed it on `main` (labelled `v56.0.0`); it is on `branch-56` but not on `branch-55`. It was requested for 55.3.0 on #26868 (https://github.com/apache/datafusion/issues/26868#issuecomment-6791317393). ## What changes are included in this PR? - A `git cherry-pick -x` of 2abf69d4a8e63fb51ed630acb7affbfb11b228ca (#25134, by @jeffw13). It applied without conflicts and this PR changes nothing else. - `datafusion/physical-plan/src/joins/sort_merge_join/bitwise_stream.rs`: `keys_match` uses the existing `JoinKeyComparator` for floating-point keys, matching the merge scan's equality (including signed zero), instead of the scalar comparator whose `partial_cmp` panics on NaN. Integer-only comparisons keep the scalar path. - `datafusion/physical-plan/src/joins/sort_merge_join/tests.rs`: `join_float_key_batch_boundaries`, 144 cases (Float16/32/64, left/right mark/semi/anti, both null-equality modes, ascending/descending, one- and two-row input batches). ## Are these changes tested? Yes, by the 144 cases that #25134 added. On this branch (Rust 1.95.0) I ran: - `cargo test -p datafusion-physical-plan --lib`: 1844 passed. - With the change in `bitwise_stream.rs` reverted and the tests kept, `cargo test -p datafusion-physical-plan --lib sort_merge_join` fails exactly the 144 new cases (96 NaN panics and 48 Float16 errors). - `cargo fmt --all -- --check` and `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` pass. ## Are there any user-facing changes? Affected sort-merge joins complete instead of failing at floating-point batch boundaries. No public API change. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
