Stefan Savic created SPARK-60338:
------------------------------------
Summary: Insert Variant canonicalization into aggregation and
repartitioning plans
Key: SPARK-60338
URL: https://issues.apache.org/jira/browse/SPARK-60338
Project: Spark
Issue Type: Sub-task
Components: Spark Core
Affects Versions: 5.0.0
Reporter: Stefan Savic
Implement optimizer rules modeled on InsertMapSortInAggregate and
InsertMapSortInRepartitionExpressions that:
• Insert CanonicalizeVariant around Variant grouping keys and distinct
aggregate arguments.
• Run the aggregation rewrite after PullOutGroupingExpressions.
• Update grouping and output references to use the canonicalized keys.
• Canonicalize explicit repartitioning keys consistently.
• Add plan tests, including extraction expressions such as
canon(variant_get(...)).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
