[
https://issues.apache.org/jira/browse/SPARK-60618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60618:
-----------------------------------
Labels: pull-request-available (was: )
> Skip the pass-through GroupPartitionsExec in a storage-partitioned join's
> alignment
> -----------------------------------------------------------------------------------
>
> Key: SPARK-60618
> URL: https://issues.apache.org/jira/browse/SPARK-60618
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> EnsureRequirements.checkKeyGroupCompatible builds a GroupPartitionsExec for
> both sides of a co-partitioned join when it pushes the merged partition keys.
> A side that was shuffled onto the other side's layout inherits that same
> projection, so its node only re-states the partitioning its shuffle already
> reports: it maps each partition onto itself, coalesces nothing, and reports
> the same keys, marker and routing. Downstream rules then have to look through
> a node that carries no information. ConvertSortMergeJoinToShuffledHashJoin,
> for one, does not reach the shuffle stage behind it, so a sort merge join
> that could be a shuffled hash join keeps its child sorts - on a MERGE whose
> target side spills terabytes, that is the dominant cost.
> This skips that node: a side is regrouped only when the GroupPartitionsExec
> it would get reports something other than the child's partitioning. Equality
> also pins the mapping itself (the keys are the child's, in the child's order)
> and covers the layout's marker and routing, so a node that coalesces several
> splits per key, pads missing keys, prunes, reduces keys, or
> spreads/replicates is still built.
> Tests: the assertions that required the node on both sides now read the same
> fact from the side's own partitioning.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
