XiDuo You created SPARK-60674:
---------------------------------
Summary: Convert a sort merge join to a shuffled hash join through
a GroupPartitionsExec
Key: SPARK-60674
URL: https://issues.apache.org/jira/browse/SPARK-60674
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
ConvertSortMergeJoinToShuffledHashJoin reaches a join side's shuffle stage only
through the join's own sort. A GroupPartitionsExec that the co-partitioned
alignment keeps over that stage (a partially clustered pairing keeps its nodes,
and a side of such a pairing can be a keyed shuffle stage) stops the traversal,
so a sort merge join that could be a shuffled hash join keeps its child sorts
even though the build side fits a local hash map.
The rule now looks through GroupPartitionsExec as well, and folds the shuffle's
per-partition bytes the way its grouping regroups them: an output partition
holds every child partition in its group, so the build-size bound is the
largest group sum rather than the largest child partition. Nodes that coalesce,
pad, prune, reduce or replicate keep their own reported partitioning and are
handled unchanged by the rest of the rule; only the bound they imply changes.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
