entity-0-0-2 opened a new pull request, #26146:
URL: https://github.com/apache/datafusion/pull/26146
## Pain
An equi-join or range join whose key is a nested type containing float
leaves can silently return fewer rows than it should. No error is raised: the
query plans, runs, and hands back a table that is missing matched pairs. With
`prefer_hash_join = false` (the sort-merge path):
```sql
CREATE TABLE fa AS SELECT * FROM (VALUES
(1, make_array(arrow_cast(-0.0, 'Float64'), 1.0)),
(2, make_array(0.0, 0.0)),
(3, make_array(0.0, 1.0))) t(id, k);
CREATE TABLE fb AS SELECT * FROM (VALUES
(10, make_array(0.0, 1.0)),
(11, make_array(0.0, 0.0))) t(id, k);
SET datafusion.optimizer.prefer_hash_join = false;
SELECT fa.id, fb.id FROM fa JOIN fb ON fa.k = fb.k;
```
DataFusion returns rows `(1,10)` and `(3,10)`; the match `(2,11)` is
dropped. Hash join returns the correct three rows.
## Root cause
`JoinKeyComparator::build`
(datafusion/physical-plan/src/joins/utils.rs:2568) normalizes `-0.0` to `+0.0`
on both key arrays before `make_comparator`, so merge-join key comparison
honors SQL equality. The sort that feeds the merge orders keys with Arrow's
IEEE total order, where `-0.0 < +0.0`. Since #24827 (d42cd854d)
`normalize_float_zero` recurses into nested children, so the two orders diverge
exactly when a key type is nested with float leaves: sorted input ranks `[-0.0,
1.0] < [0.0, 0.0]` while the normalized comparator ranks it `>`, and the merge
stream skips valid pairs. Flat float keys are unaffected because nothing sorts
between the two zeros.
## The fix
The planner routes joins with such keys away from the sort-based execs:
- datafusion/core/src/physical_planner.rs: equi-joins whose keys are nested
with float leaves now plan `HashJoinExec` instead of `SortMergeJoinExec`, and
single range predicates on such keys plan `NestedLoopJoinExec` instead of
`PiecewiseMergeJoinExec` (same fallback the planner already uses when a range
predicate has no clean side split).
- Key types where both orders provably agree, including flat floats, keep
the sort-based fast paths.
## Test
Added the issue query to
`datafusion/sqllogictest/test_files/negative_zero.slt` asserting the correct
three rows.
Before, on main (`a23b89ff9`):
```
query result mismatch:
[SQL] SELECT fa.id, fb.id FROM fa JOIN fb ON fa.k = fb.k ORDER BY fa.id,
fb.id;
[Diff] (-expected|+actual)
1 10
- 2 11
3 10
```
After: `cargo test -p datafusion-sqllogictest --test sqllogictests --
negative_zero` passes, as do the `sort_merge_join*` and `piecewise_merge_join*`
suites and `cargo test -p datafusion --lib physical_planner` (56 passed).
## How this was found
Vestiges Causal Walk (https://github.com/samvallad33/vestige). There was no
error to search for, only wrong bytes and a walk: the symptom record (the
missing `2 11` row) was bridged over recorded causal edges to the introducing
commit `d42cd854d`.
Fixes #26115
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
