jayzhan211 opened a new issue, #26496:
URL: https://github.com/apache/datafusion/issues/26496
### Describe the bug
An uncorrelated null-aware anti join (`build.c1 NOT IN (probe.c2)`,
`JoinType::LeftAnti` with `null_aware = true`) built with
`PartitionMode::Partitioned` returns rows even when the probe side contains a
NULL. SQL semantics say that if the subquery produces a NULL, `x NOT IN (...)`
is never true, so the result must be empty. The same join in
`PartitionMode::CollectLeft` returns the correct empty result.
The cause is that in `Partitioned` mode every output partition builds its
own build side, and the "probe side saw a NULL" fact is recorded on that
per-partition build data (`hash_join/exec.rs` around line 1935,
`hash_join/stream.rs` around line 1366). A NULL probe key is hash-routed to one
partition, so it only suppresses the build rows of that partition. The others
still emit their unmatched rows.
`NullAwareMode::try_new` already rejects `Partitioned` for the correlated
`LeftAnti` and `LeftMark` forms, but accepts it for the uncorrelated `LeftAnti`
form.
### To Reproduce
The DataFusion SQL planner never produces this plan: `JoinSelection` and the
physical planner both pin null-aware joins to `CollectLeft`. It is reachable
from plans built by hand or deserialized from protobuf (`HashJoinExecNode`
decodes the partition mode as written).
Build side `c1` = 1..8, probe side `c2` = {1, NULL}, both sides
`RepartitionExec(Hash([c], 4))`, `HashJoinExec` with `JoinType::LeftAnti`,
`PartitionMode::Partitioned`, `null_aware = true`, `NullEqualsNothing`.
### Expected behavior
No rows (as `CollectLeft` returns).
### Actual behavior
`[3, 6, 8]` — the build rows hashed to partitions that did not receive the
NULL probe row.
### Additional context
A fix that rejects this shape at plan construction, consistent with the
existing rejection for the correlated forms, is ready; I'll link the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
