phillipleblanc opened a new issue, #2187: URL: https://github.com/apache/datafusion-ballista/issues/2187
## Bug SQL `NOT IN` predicates are planned as null-aware `LeftAnti` hash joins. DataFusion requires these joins to: - remain `LeftAnti` (they cannot be swapped to `RightAnti`), and - run in `PartitionMode::CollectLeft` so every probe partition observes the build side's global NULL state. Ballista's scheduler can currently violate both invariants: 1. `DefaultDistributedPlanner::maybe_promote_to_broadcast` checks generic `CollectLeft` broadcast safety before its `null_aware` exception. Because generic `LeftAnti` is not broadcast-safe, it demotes a null-aware `CollectLeft` join to `Partitioned` despite the adjacent comment saying null-aware joins are never demoted. 2. Ballista's copied `JoinSelection` can swap an already-partitioned null-aware join in `statistical_join_selection_subrule`. 3. Its unbounded-input `hash_join_swap_subrule` is missing DataFusion's `!hash_join.null_aware` guard. 4. The AQE resolver applies generic size-based swapping and broadcast-safety rules to null-aware joins, allowing them to be swapped or repartitioned. A swap is rejected while resolving stages with an error similar to: ```text null_aware can only be true for LeftAnti joins, got RightAnti ``` The task-status update then fails and the distributed query can remain running until the client times out. Leaving the join partitioned can instead produce incorrect `NOT IN` NULL semantics. ## Expected behavior All static and adaptive scheduler planning paths should preserve null-aware joins as unswapped `LeftAnti(CollectLeft)` joins, independently of normal broadcast thresholds. ## Proposed fix - Check `null_aware` before generic `CollectLeft` demotion. - Route already-partitioned null-aware joins through the existing helper that rebuilds them as `CollectLeft`. - Match DataFusion's null-aware guard in the unbounded-input swap rule. - Make AQE force `CollectLeft` and disable size-driven swapping for null-aware joins. - Add regression coverage for the static, unbounded-input, and AQE paths. This was originally observed and fixed downstream in spiceai/datafusion-ballista#58; this issue tracks a fresh implementation against current Apache `main`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
