[
https://issues.apache.org/jira/browse/SPARK-60741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
XiDuo You reassigned SPARK-60741:
---------------------------------
Assignee: XiDuo You
> Keep the build side's keyed partitioning in the dynamic partition pruning
> subquery
> ----------------------------------------------------------------------------------
>
> Key: SPARK-60741
> URL: https://issues.apache.org/jira/browse/SPARK-60741
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: XiDuo You
> Assignee: XiDuo You
> Priority: Major
> Labels: pull-request-available
>
> When the build side of dynamic partition pruning is a
> storage-partitioned-join scan, the subquery that collects the build-side join
> keys groups by aliases of those keys. The physical partial aggregate replaces
> each grouping alias with a fresh attribute, so the alias-aware output
> partitioning cannot project the scan's KeyedPartitioning through it and
> reports UnknownPartitioning. EnsureRequirements then inserts a hash shuffle
> between the partial and final aggregate, even though the scan's keyed
> partitioning could satisfy the clustering after a GroupPartitionsExec
> coalesce.
> This change aliases the keys in a Project below the aggregate and groups by
> the resulting attributes, so the aggregate keeps the build side's
> partitioning. It applies to PlanDynamicPruningFilters and
> PlanAdaptiveDynamicPruningFilters.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
