saadtajwar opened a new pull request, #23957: URL: https://github.com/apache/datafusion/pull/23957
## Which issue does this PR close? - Part of #23931 (not closes yet) ## Rationale for this change (copied from issue): DataFusion plans NOT IN (subquery) as a null-aware anti join, but HashJoinExec only supports null_aware = true for LeftAnti with a single join key (validated in datafusion-physical-plan/src/joins/hash_join/exec.rs). Since HashJoinExec always builds on the left input, the build side of a null-aware anti join is the outer table, not the subquery. This has two costs: Memory scales with the wrong side. For SELECT ... FROM big_fact WHERE key NOT IN (SELECT k FROM small_dim), the hash table is built over the entire fact table. Memory is O(outer) when it could be O(subquery). The operator cannot be distributed. The null-aware logic coordinates three pieces of global state across probe partitions through in-process shared memory: probe_side_has_null: AtomicBool, probe_side_non_empty: AtomicBool, and the visited-build-row bitmap, with the last probe partition to finish emitting the unmatched build rows (hash_join/stream.rs). This is correct and cheap in one process, but engines that split probe partitions across processes get independent copies of all three and produce duplicated or incorrect rows. Ballista hit exactly this (https://github.com/apache/datafusion-ballista/issues/2187) and currently has to force the join into a single task (https://github.com/apache/datafusion-ballista/pull/2188), losing all parallelism. ## What changes are included in this PR? This PR is the first of two - when a path IS built as `RightAnti` + null aware + `CollectLeft` partition mode, the join executes as expected. The second PR that will be created after this will support the planner changes to emit null-aware `RightAnti` & the `JoinSelection` swap logic! This PR specifically adds the `build_side_has_null` field and assigns at build-time, and correctly satisfies the invariant of no rows being output if true + build state with null keys not being output + empty build side outputting all probe rows ## Are these changes tested? Yes ## Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
