adriangb opened a new pull request, #25560: URL: https://github.com/apache/datafusion/pull/25560
## Which issue does this PR close? - Completes https://github.com/apache/datafusion/issues/25336 for the mark-join shapes. > [!IMPORTANT] > **Stacked on https://github.com/apache/datafusion/pull/25558 and https://github.com/apache/datafusion/pull/25559.** GitHub cannot base a PR on another fork's branch, so the diff here shows all three. **Review only the last commit** (`fix: request a null-aware mark join wherever a NULL mark is observable`). Draft until the other two land; I will rebase then. ## Rationale for this change Decorrelation asked for null-aware semantics only for some `NOT IN` mark joins. A non-equality correlation leaves a residual join filter rather than an equi-join key, and that shape was planned without it, so a NULL on the subquery side read as FALSE instead of UNKNOWN: ```sql CREATE TABLE oc(id INT, g INT, h INT) AS VALUES (2,2,0),(1,2,0),(9,2,5),(3,1,0); CREATE TABLE ic(id INT, k INT) AS VALUES (1,1),(NULL,1),(2,2); SELECT id FROM oc WHERE oc.h > 1 OR oc.id NOT IN (SELECT i.id FROM ic i WHERE i.k < oc.g) ORDER BY id; ``` returns `2, 3, 9`; DuckDB returns `3, 9`. Row `id = 2` is UNKNOWN, not TRUE. #25559 gives the executor the machinery. This asks for it in the remaining place. ## What changes are included in this PR? A mark join needs null-aware semantics only where a NULL mark can behave differently from a FALSE mark. `AND`/`OR` give TRUE only from TRUE, and a `Filter` keeps a row only when the predicate is TRUE, so a non-negated `IN` reached through nothing but `AND`/`OR` is identical with a FALSE mark and must stay on the plain join. Requesting it there is expensive: **1–2 ms → 7.4 s at 100k × 100k**. That test is made **per subquery occurrence**, in a recursion that only knows `AND`/`OR`. The permissive outcome lives in one arm whose pattern is its own proof — a non-negated `IN` whose value holds no subquery, reached through nothing but `AND`/`OR` frames — and everything else, including future `Expr` variants, takes the null-aware branch. Two further changes in `build_join`: - a constant `IN` value is projected as an outer column so the equality becomes `on[0]`, the key position the executor reads as the `NOT IN` value key. Otherwise a correlation takes that slot and the value-key NULL rules are applied to the wrong key; - `null_aware` is computed once instead of being re-derived for the constant projection, the mark branch and the anti branch. Net effect on the optimizer source is **−14 lines**. ## What is the testing strategy for this PR? The coverage landed in #25558; this PR flips the remaining expectations, including the Q08 canary, so the diff shows the behaviour change. The plan pins added in #25558 guard the *negative* direction — that a positive `IN` stays on the plain join — which no result assertion can catch, since a needless null-aware join is correct, only slower. Mutation testing over the decision confirms it: mutants that widen null-awareness are killed only by those pins. ## Are there any user-facing changes? Correlated `NOT IN` inside a larger predicate returns correct results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
