XiDuo You created SPARK-60741:
---------------------------------
Summary: Keep the build side's keyed partitioning in the dynamic
partition pruning subquery
Key: SPARK-60741
URL: https://issues.apache.org/jira/browse/SPARK-60741
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 5.0.0
Reporter: XiDuo You
When the build side of dynamic partition pruning is a storage-partitioned-join
scan, the subquery that collects the build-side join keys groups by aliases of
those keys. The physical partial aggregate replaces each grouping alias with a
fresh attribute, so the alias-aware output partitioning cannot project the
scan's KeyedPartitioning through it and reports UnknownPartitioning.
EnsureRequirements then inserts a hash shuffle between the partial and final
aggregate, even though the scan's keyed partitioning could satisfy the
clustering after a GroupPartitionsExec coalesce.
This change aliases the keys in a Project below the aggregate and groups by the
resulting attributes, so the aggregate keeps the build side's partitioning. It
applies to PlanDynamicPruningFilters and PlanAdaptiveDynamicPruningFilters.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
