[
https://issues.apache.org/jira/browse/SPARK-57023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kent Yao resolved SPARK-57023.
------------------------------
Fix Version/s: 5.0.0
Resolution: Fixed
Issue resolved by pull request 56078
[https://github.com/apache/spark/pull/56078]
> Peel scale-preserving widening decimal Cast in front of MIN/MAX
> ---------------------------------------------------------------
>
> Key: SPARK-57023
> URL: https://issues.apache.org/jira/browse/SPARK-57023
> Project: Spark
> Issue Type: Improvement
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Kent Yao
> Assignee: Kent Yao
> Priority: Minor
> Labels: decimal, optimizer, pull-request-available
> Fix For: 5.0.0
>
>
> h3. Problem
> When a decimal aggregate input is wrapped by a widening, scale-preserving
> {{Cast}} (e.g. {{Cast(d as Decimal(p+k, s))}} where {{k >= 0}} and scale
> unchanged), {{MIN}}/{{MAX}} computes the result in the wider type even though
> the cast is order-preserving and the min/max value is bit-identical in the
> narrower type. This bloats codegen / shuffle payload and wastes a
> {{Decimal.changePrecision}} per row.
> {{SUM}} and {{AVG}} already peel this pattern in {{DecimalAggregates}} (see
> SPARK-3933 for the original arm and SPARK-56983 for the {{evalMode}}-aware
> follow-up). {{MIN}}/{{MAX}} were never extended.
> h3. Proposal
> Extend {{DecimalAggregates}} with a {{MIN}}/{{MAX}} arm that peels the
> widening {{Cast}} when:
> * child precision/scale fit, and
> * {{evalMode}} of the surrounding aggregate is preserved (mirrors SPARK-56983
> semantics).
> Refactor the existing SUM/AVG arms to share a {{WidenedDecimalChild}}
> extractor (private object) so the new MIN/MAX arm reuses the predicate.
> h3. Scope (non-goals)
> * No changes to ANSI overflow semantics — peel is bit-identical only when
> scale is preserved and child precision fits.
> * No new SQL surface, no new SQLConf.
> h3. References
> * SPARK-3933 — original DecimalAggregates SUM/AVG peel
> * SPARK-56983 — evalMode-preserving variant (sibling implementation)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
