[ 
https://issues.apache.org/jira/browse/SPARK-56627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kent Yao updated SPARK-56627:
-----------------------------
    Affects Version/s: 4.2.0
                           (was: 4.1.0)
          Description: 
## Background

The Catalyst optimizer rule `DecimalAggregates` (in 
`org.apache.spark.sql.catalyst.optimizer`) rewrites `Sum(d : DECIMAL(p, s))` 
with `p + 10 <= MAX_LONG_DIGITS (=18)` into `MakeDecimal(Sum(UnscaledValue(d)), 
p + 10, s)`, replacing the generic `CheckOverflow` decimal path with a 
Long-arithmetic fast-path that materially reduces per-row cost on aggregate and 
window paths.

This rewrite is bypassed today when the input is wrapped in a *scale-preserving 
widening* Cast, e.g. `Sum(Cast(d : DECIMAL(p, s) AS DECIMAL(p* > p`, `s* == s` 
(scale preserved), and the inner `Sum(child)` would satisfy the existing `p + 
10 <= MAX_LONG_DIGITS` precondition.
2. Peels the Cast and lets the existing fast-path rewrite fire on the inner 
`Sum(child)`.
3. Is gated by a new internal SQLConf, default `false`, so existing plans are 
byte-identical with the conf off.

Scope:

* Phase 1: `Sum` only.
* Aggregate arm and Window arm covered by a single peel point.
* Rule plumbing only — no public API change, no SQL grammar change, no metrics 
change.

## SQLConf

* Key: `spark.sql.optimizer.decimalAggregates.peelWidenedCastForSum.enabled`
* Type: `boolean`
* Default: `false`
* Visibility: `.internal()`

## Behaviour change

* SQLConf default (`false`): byte-identical to today.
* SQLConf `true`: only the qualifying `Sum(Cast(...))` shape changes.

## Verification

* Plan-shape PBT in `DecimalAggregatesSuite` (oracle-1/oracle-2/idempotence).
* Numerical-equivalence PBT in `DataFrameAggregateSuite`.
* `DecimalAggregatesBenchmark` under `sql/core`. Local JDK 17 smoke (N=1M, 
iters=2): peel-on per-row ns < peel-off on all 6 cases at both 
`decimalOperations.allowPrecisionLoss=true/false`. A1 ~1.2-1.3x.

## Public API change

None.

## SPIP

Not required.

## Affects Version

4.2.0 (master).

  was:
## Background

The Catalyst optimizer rule `DecimalAggregates` (in 
`org.apache.spark.sql.catalyst.optimizer`) rewrites `Sum(d : DECIMAL(p, s))` 
with `p + 10 <= MAX_LONG_DIGITS (=18)` into `MakeDecimal(Sum(UnscaledValue(d)), 
p + 10, s)`, replacing the generic `CheckOverflow` decimal path with a 
Long-arithmetic fast-path that materially reduces per-row cost on aggregate and 
window paths.

This rewrite is bypassed today when the input is wrapped in a *scale-preserving 
widening* Cast, e.g. `Sum(Cast(d : DECIMAL(p, s) AS DECIMAL(p', s)))` with `p' 
> p` and matching scale. The Cast is opaque to the rule's pattern, so the 
optimizer keeps the slower `CheckOverflow` path even though the cast is 
provably value-preserving and the inner `Sum(d)` would otherwise satisfy the 
precondition `p + 10 <= 18`.

This pattern shows up naturally in user queries that pre-cast a narrow decimal 
column to a wider decimal before aggregation, and in plans where another rule 
has inserted the widening Cast (e.g. `PromoteStrings`, type-coercion lifts).

## Proposal

Add a peel step inside `DecimalAggregates` that:

1. Recognises `Sum(Cast(child, DECIMAL(p', s')))` where the child has type 
`DECIMAL(p, s)`, `p' > p`, `s' == s` (scale preserved), and the inner 
`Sum(child)` would satisfy the existing `p + 10 <= MAX_LONG_DIGITS` 
precondition.
2. Peels the Cast and lets the existing fast-path rewrite fire on the inner 
`Sum(child)`.
3. Is gated by a new internal SQLConf, default `false`, so existing plans are 
byte-identical with the conf off.

Scope:

* Phase 1: `Sum` only. AVG and other aggregates are out of scope (tracked 
separately if needed).
* Aggregate arm and Window arm of `Sum` are both covered by a single peel point 
in the rule.
* Rule plumbing only — no public API change, no SQL grammar change, no metrics 
change.

## SQLConf

* Key: `spark.sql.optimizer.decimalAggregates.peelWidenedCastForSum.enabled`
* Type: `boolean`
* Default: `false`
* Visibility: `.internal()`
* Docstring: gated peel step inside `DecimalAggregates`; rollback knob.

## Behaviour change

* With SQLConf default (`false`): byte-identical to today on every plan. No row 
count, no result, no plan shape difference.
* With SQLConf `true`: only the qualifying `Sum(Cast(...))` shape changes — 
gains the existing fast-path. Numerical equivalence covered by property-based 
tests in `DataFrameAggregateSuite` (oracle-1/2/4 fixed witnesses).

## Verification

* Plan-shape property-based tests (catalyst): `DecimalAggregatesSuite` — 
oracle-1 (peel fires when contract holds) + oracle-2 (peel does not fire 
outside contract) + idempotence on `plan.canonicalized`.
* Numerical-equivalence property-based tests (sql-core): 
`DataFrameAggregateSuite` — random witness sweep + fixed `(p=8, p'=30, s=2)` 
witness.
* Benchmark: new `DecimalAggregatesBenchmark` under 
`sql/core/src/test/scala/.../execution/benchmark/`. Local JDK 17 smoke (AMD 
EPYC 7763, N=1M, iters=2): peel-on per-row ns strictly less than peel-off on 
all 6 cases (4 Aggregate + 2 Window) at both 
`spark.sql.decimalOperations.allowPrecisionLoss=true` and `=false`. A1 (p=7, 
s=2, p'=17) shows ~1.2-1.3x. Multi-JDK GHA results will be attached on PR open.

## Public API change

None.

## SPIP

Not required. Internal optimizer rule, gated by an internal SQLConf default 
`false`. No public API, no semantic change, no grammar change.

## Rollback

Disable the SQLConf.

## Affects Version

4.1.0 (master).


> DecimalAggregates: peel scale-preserving widening Cast for Sum to enable 
> fast-path rewrite
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-56627
>                 URL: https://issues.apache.org/jira/browse/SPARK-56627
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 4.2.0
>            Reporter: Kent Yao
>            Priority: Major
>
> ## Background
> The Catalyst optimizer rule `DecimalAggregates` (in 
> `org.apache.spark.sql.catalyst.optimizer`) rewrites `Sum(d : DECIMAL(p, s))` 
> with `p + 10 <= MAX_LONG_DIGITS (=18)` into 
> `MakeDecimal(Sum(UnscaledValue(d)), p + 10, s)`, replacing the generic 
> `CheckOverflow` decimal path with a Long-arithmetic fast-path that materially 
> reduces per-row cost on aggregate and window paths.
> This rewrite is bypassed today when the input is wrapped in a 
> *scale-preserving widening* Cast, e.g. `Sum(Cast(d : DECIMAL(p, s) AS 
> DECIMAL(p* > p`, `s* == s` (scale preserved), and the inner `Sum(child)` 
> would satisfy the existing `p + 10 <= MAX_LONG_DIGITS` precondition.
> 2. Peels the Cast and lets the existing fast-path rewrite fire on the inner 
> `Sum(child)`.
> 3. Is gated by a new internal SQLConf, default `false`, so existing plans are 
> byte-identical with the conf off.
> Scope:
> * Phase 1: `Sum` only.
> * Aggregate arm and Window arm covered by a single peel point.
> * Rule plumbing only — no public API change, no SQL grammar change, no 
> metrics change.
> ## SQLConf
> * Key: `spark.sql.optimizer.decimalAggregates.peelWidenedCastForSum.enabled`
> * Type: `boolean`
> * Default: `false`
> * Visibility: `.internal()`
> ## Behaviour change
> * SQLConf default (`false`): byte-identical to today.
> * SQLConf `true`: only the qualifying `Sum(Cast(...))` shape changes.
> ## Verification
> * Plan-shape PBT in `DecimalAggregatesSuite` (oracle-1/oracle-2/idempotence).
> * Numerical-equivalence PBT in `DataFrameAggregateSuite`.
> * `DecimalAggregatesBenchmark` under `sql/core`. Local JDK 17 smoke (N=1M, 
> iters=2): peel-on per-row ns < peel-off on all 6 cases at both 
> `decimalOperations.allowPrecisionLoss=true/false`. A1 ~1.2-1.3x.
> ## Public API change
> None.
> ## SPIP
> Not required.
> ## Affects Version
> 4.2.0 (master).



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to