shehabgamin opened a new pull request, #27227: URL: https://github.com/apache/datafusion/pull/27227
## Which issue does this PR close? - Part of #XXXXX. ## Rationale for this change A null-aware mark join always builds on the outer side, because there is no null-aware `RightMark`. A correlated `NOT IN` always collects one whole side, because null-aware joins reject `PartitionMode::Partitioned`. ```sql SELECT id, id IN (SELECT id FROM small) AS m FROM big SELECT * FROM a WHERE a.x NOT IN (SELECT b.y FROM b WHERE b.k = a.k) ``` ## What changes are included in this PR? Null-aware `RightMark` is now allowed. It reuses the per-probe-row logic of null-aware `RightAnti` but emits every probe row with a TRUE, FALSE or NULL mark. `JoinSelection` can swap a null-aware `LeftMark` into a `RightMark`. Null-aware joins with correlation scope keys can run `Partitioned`, with both inputs hash partitioned on the scope keys only. All rows of a scope, NULLs included, then share a partition. Joins without scope keys still need `CollectLeft`. The planner and `JoinSelection` pick this mode for large inputs. Key reordering and dynamic filters are skipped for these joins. Only `HashJoinExec` changes. `SortMergeJoinExec` keeps its narrow `null_aware` support, and the physical planner never sends null-aware joins to other join types. Before this change, `JoinSelection` could not swap a null-aware `LeftMark` because there was no null-aware `RightMark`, so the outer side was always the build side. Now it swaps them like any other join and builds on the smaller input. That is why several `EXPLAIN` plans change from `LeftMark` to `RightMark`. Their results do not change. This comes with a tradeoff that `JoinSelection` already makes for other join types. A `RightMark` keeps every outer row, so it gives up the dynamic filter that a `LeftMark` pushes into the subquery side. The swap only happens when the subquery side is the smaller input, so the filter it loses would have pruned the smaller side anyway. ## What is the testing strategy for this PR? Unit tests for `RightMark` and for `Partitioned` against `CollectLeft`, a `JoinSelection` test, fuzz test plan checks, and two new sqllogictests, `null_aware_right_mark_join.slt` and `null_aware_partitioned_join.slt`. ## Are there any user-facing changes? No -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
