[ 
https://issues.apache.org/jira/browse/SPARK-61860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

XiDuo You updated SPARK-61860:
------------------------------
    Summary: Retarget the hash shuffle below a clustered operator to the keyed 
layout  (was: Push a keyed shuffle down through clustered operators in AQE 
preparation)

> Retarget the hash shuffle below a clustered operator to the keyed layout
> ------------------------------------------------------------------------
>
>                 Key: SPARK-61860
>                 URL: https://issues.apache.org/jira/browse/SPARK-61860
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: XiDuo You
>            Priority: Major
>              Labels: pull-request-available
>
> AQE preparation gains a small rule that removes one shuffle from a 
> storage-partitioned join whose shuffled side aggregates or windows on the 
> join keys.
> With spark.sql.sources.v2.bucketing.shuffle.enabled, EnsureRequirements plans 
> two independent shuffles on the shuffled side: a hash shuffle under the 
> aggregate (or window) and a keyed shuffle above it that lays the aggregate's 
> output out on the storage-partitioned side's keys. A keyed layout that serves 
> the aggregate's clustering serves it as well as the hash layout did, so the 
> rule retargets the hash shuffle to the keyed layout and drops the keyed 
> shuffle above the operator.
> The walk crosses grouped aggregates, partitioned windows and final window 
> group limits whose keys the layout clusters, plus deterministic projections, 
> filters and local sorts. It is gated like 
> KeyedShuffleSpec.canCreatePartitioning (one-side shuffle on, partially 
> clustered off), keeps both shuffles if the plan reuses either, and validates 
> the whole plan against the root distribution and recorded user repartitions.
> Scope: master. Internal config 
> spark.sql.adaptive.pushDownKeyedShuffle.enabled, plus an opt-in 
> allowWithSubqueries config because a rewrite inside a subquery can cost the 
> query an exchange reuse.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to