vanekjar opened a new pull request, #57934: URL: https://github.com/apache/spark/pull/57934
### What changes were proposed in this pull request? Add `ExistenceJoin` to `canSplitLeftSide` in `OptimizeSkewedJoin`, so AQE can split a skewed left (probe) side of an existence join. The right side stays ineligible, exactly as it is for `LeftSemi` / `LeftAnti`. This re-proposes PR #42003 (same JIRA, bot-closed as stale in Nov 2023) ### Why are the changes needed? `RewritePredicateSubquery` turns a top-level `IN` / `EXISTS` into a `LeftSemi` join and a top-level `NOT IN` / `NOT EXISTS` into `LeftAnti`, both of which AQE already skew-splits. A subquery nested inside a larger expression, in practice a disjunction, e.g. `WHERE v > 0 OR k IN (SELECT ...)`, instead goes through `rewriteExistentialExpr` and becomes an `ExistenceJoin`. Since `ExistenceJoin` is in neither `canSplitLeftSide` nor `canSplitRightSide`, `OptimizeSkewedJoin` never splits it, and a skewed probe-side key runs as a single straggler task no matter how parallel the rest of the stage is. Splitting the left side is safe. `ExistenceJoin` can only be built on the right, `canBuildBroadcastLeft` and `canBuildShuffledHashJoinLeft` both exclude it, so the left side is always the probe side, and each left row is processed independently (`HashJoin.existenceJoin` is a `streamIter.map`). That is the same property that makes `LeftSemi` / `LeftAnti` splittable, and `ShuffledJoin.outputPartitioning` already maps `LeftExistence(_)` to `left.outputPartitioning` for all three. ### Does this PR introduce _any_ user-facing change? No. Query results are unchanged. With AQE and skew join optimization enabled, the skewed probe side of an existence join is now split across multiple tasks instead of running in one. ### How was this patch tested? New unit test "SPARK-44426: adaptive skew join for ExistenceJoin" in `AdaptiveQueryExecSuite`. It reuses the SPARK-29544 fixture, in which both sides are skewed, so a single query asserts both that the left side is split and that the right side is left alone while being genuinely eligible on size. Also verified end to end in our production workload to make sure the skew was resolved. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
