XiDuo You created SPARK-60618:
---------------------------------
Summary: Skip the pass-through GroupPartitionsExec in a
storage-partitioned join's alignment
Key: SPARK-60618
URL: https://issues.apache.org/jira/browse/SPARK-60618
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
EnsureRequirements.checkKeyGroupCompatible builds a GroupPartitionsExec for
both sides of a co-partitioned join when it pushes the merged partition keys. A
side that was shuffled onto the other side's layout inherits that same
projection, so its node only re-states the partitioning its shuffle already
reports: it maps each partition onto itself, coalesces nothing, and reports the
same keys, marker and routing. Downstream rules then have to look through a
node that carries no information. ConvertSortMergeJoinToShuffledHashJoin, for
one, does not reach the shuffle stage behind it, so a sort merge join that
could be a shuffled hash join keeps its child sorts - on a MERGE whose target
side spills terabytes, that is the dominant cost.
This skips that node: a side is regrouped only when the GroupPartitionsExec it
would get reports something other than the child's partitioning. Equality also
pins the mapping itself (the keys are the child's, in the child's order) and
covers the layout's marker and routing, so a node that coalesces several splits
per key, pads missing keys, prunes, reduces keys, or spreads/replicates is
still built.
Tests: the assertions that required the node on both sides now read the same
fact from the side's own partitioning.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
