[
https://issues.apache.org/jira/browse/SPARK-60379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60379:
-----------------------------------
Labels: pull-request-available (was: )
> Push down shuffle exchange through union
> ----------------------------------------
>
> Key: SPARK-60379
> URL: https://issues.apache.org/jira/browse/SPARK-60379
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> An `EnsureRequirements` shuffle that sits above a `UnionExec` shuffles the
> whole union, even
> when some of the union's children already produce a layout that answers the
> requirement, or when
> a layout one of them produces would answer it if the other children were laid
> out on it.
> Push that shuffle below the union instead: pick one of the layouts the
> union's own children
> report, keep the children that already report it, and shuffle the rest onto
> it. The operator
> above then runs with the picked layout's partition count. The plan as a whole
> is validated
> before the rewrite is taken.
> The operators between the shuffle and the union can be looked through (a
> project, a local sort,
> a filter, a partial aggregate, a partial window group limit) under a config
> of their own,
> because a child is then shuffled before the reduction, which can send more
> data than the one
> shuffle the rewrite replaces.
> The rule runs during AQE query stage preparation, so no exchange is
> materialized before it
> gets a chance to remove one.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
