sundapeng opened a new pull request, #11308:
URL: https://github.com/apache/paimon/pull/11308

   ### Purpose
   
   On a table with `merge-engine=aggregation` and 
`fields.<col>.aggregate-function='sum'`, a DECIMAL column can overflow its 
declared precision in two silent ways:
   
   1. The compact fast path in `DecimalUtils.add/subtract` checked only long 
overflow (`Math.addExact`) but not the declared precision, so `99999 + 1` on 
`DECIMAL(5,0)` returned `Decimal.fromUnscaledLong(100000, 5, 0)`: a value 
violating its own precision invariant. The corrupt value is then written to 
data files — `ParquetRowDataWriter.Int32Writer` casts `unscaledLong` to `int` 
and stores the out-of-range unscaled value under the column's declared 
precision.
   2. The non-compact path returns null from `Decimal.fromBigDecimal` when the 
result precision overflows; `FieldSumAgg` did not check it, so the aggregated 
column silently became NULL and the whole accumulated value was lost — the next 
merge restarted the sum from scratch.
   
   This adds a precision bounds check to the compact fast path (falling through 
to `fromBigDecimal`, which returns null on overflow) and throws 
`ArithmeticException` on the null result in `FieldSumAgg#agg` and `#retract`, 
following the integral-overflow precedent of #7906 and #7992.
   
   `DECIMAL` is not covered by the `fields.<field>.sum.fail-on-overflow` option 
added in #9874 (integral types only): a decimal precision overflow has no 
wrap-around semantics, so silent NULL or corruption is the only alternative to 
failing.
   
   Compatibility consequence, the same one #7906/#7992 accepted for integral 
types: jobs that today "succeed" while writing corrupt decimals will now fail 
with `ArithmeticException`.
   
   ### Tests
   
   - `DecimalTest#testAddSubtractPrecisionOverflow` (paimon-common): compact 
and non-compact `add`/`subtract` return null instead of an out-of-range 
`Decimal`, for both `DECIMAL(5,0)` and `DECIMAL(5,2)` (the bound is 
`10^precision`, not `10^(precision+scale)`). Fails without the fix: `expected: 
null but was: 100000`.
   - `FieldAggregatorTest#testFieldSumDecimalOverflow` / 
`#testFieldSumDecimalRetractOverflow` (compact, `DECIMAL(5,0)`), 
`#testFieldSumDecimalOverflowWithScale` (compact, `DECIMAL(5,2)`) and 
`#testFieldSumDecimalOverflowNonCompact` (`DECIMAL(38,2)`): `agg`/`retract` 
throw `ArithmeticException` on overflow. All fail without the fix.
   - Existing `DecimalTest` cases exercising `add()` and the full 
`FieldAggregatorTest` pass unchanged.
   
   ### API and Format
   
   No.
   
   ### Documentation
   
   No. The `fail-on-overflow` docs added in #9874 cover integral types only; a 
decimal precision overflow has no ignore mode to document.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to