ewoodbury opened a new pull request, #2898: URL: https://github.com/apache/datafusion-ballista/pull/2898
## Which issue does this PR close? Closes #2849. ## Rationale for this change A `NOT IN` subquery that stays a null-aware anti join has to read the whole subquery result in one task to be correct. The static planner gets that by putting a coalesce in front of the join, but when there's an aggregate above the join, the planner later removes that coalesce. The join then runs as several tasks, and each one returns its own copy of the rows it didn't match. Example: with the static planner and the `NOT IN` rewrite turned off, this query over a 4-file `t2` returns 26/26/23 instead of 5/5/5: ```sql select a % 3, count(*) from t1 where a not in (select b from t2) group by a % 3 ``` It only happens with the static planner (or pipelined mode, which always uses it). Default settings don't hit it. ## What changes are included in this PR? The static planner now ends the stage right after a null-aware join, so nothing planned above it can remove the coalesce. The subquery side is still read in parallel. The cost is one extra stage, carrying the join's output, which is never larger than the table the join already holds in memory. A one-partition coalesced shuffle reader would avoid the extra stage, but the static planner doesn't support those yet, so that's left as a follow-up. Tests: the `null_aware` client test gets a static-planner case and a `GROUP BY` query (that combination fails on main), and a planner test checks the new stage. ## Are there any user-facing changes? Correct results. `EXPLAIN` shows one more stage for null-aware anti joins under the static planner. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
