[
https://issues.apache.org/jira/browse/SPARK-61860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You updated SPARK-61860:
------------------------------
Summary: Retarget the hash shuffle below a clustered operator to the keyed
layout (was: Push a keyed shuffle down through clustered operators in AQE
preparation)
> Retarget the hash shuffle below a clustered operator to the keyed layout
> ------------------------------------------------------------------------
>
> Key: SPARK-61860
> URL: https://issues.apache.org/jira/browse/SPARK-61860
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> AQE preparation gains a small rule that removes one shuffle from a
> storage-partitioned join whose shuffled side aggregates or windows on the
> join keys.
> With spark.sql.sources.v2.bucketing.shuffle.enabled, EnsureRequirements plans
> two independent shuffles on the shuffled side: a hash shuffle under the
> aggregate (or window) and a keyed shuffle above it that lays the aggregate's
> output out on the storage-partitioned side's keys. A keyed layout that serves
> the aggregate's clustering serves it as well as the hash layout did, so the
> rule retargets the hash shuffle to the keyed layout and drops the keyed
> shuffle above the operator.
> The walk crosses grouped aggregates, partitioned windows and final window
> group limits whose keys the layout clusters, plus deterministic projections,
> filters and local sorts. It is gated like
> KeyedShuffleSpec.canCreatePartitioning (one-side shuffle on, partially
> clustered off), keeps both shuffles if the plan reuses either, and validates
> the whole plan against the root distribution and recorded user repartitions.
> Scope: master. Internal config
> spark.sql.adaptive.pushDownKeyedShuffle.enabled, plus an opt-in
> allowWithSubqueries config because a rewrite inside a subquery can cost the
> query an exchange reuse.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]