sundapeng opened a new pull request, #11308: URL: https://github.com/apache/paimon/pull/11308
### Purpose On a table with `merge-engine=aggregation` and `fields.<col>.aggregate-function='sum'`, a DECIMAL column can overflow its declared precision in two silent ways: 1. The compact fast path in `DecimalUtils.add/subtract` checked only long overflow (`Math.addExact`) but not the declared precision, so `99999 + 1` on `DECIMAL(5,0)` returned `Decimal.fromUnscaledLong(100000, 5, 0)`: a value violating its own precision invariant. The corrupt value is then written to data files — `ParquetRowDataWriter.Int32Writer` casts `unscaledLong` to `int` and stores the out-of-range unscaled value under the column's declared precision. 2. The non-compact path returns null from `Decimal.fromBigDecimal` when the result precision overflows; `FieldSumAgg` did not check it, so the aggregated column silently became NULL and the whole accumulated value was lost — the next merge restarted the sum from scratch. This adds a precision bounds check to the compact fast path (falling through to `fromBigDecimal`, which returns null on overflow) and throws `ArithmeticException` on the null result in `FieldSumAgg#agg` and `#retract`, following the integral-overflow precedent of #7906 and #7992. `DECIMAL` is not covered by the `fields.<field>.sum.fail-on-overflow` option added in #9874 (integral types only): a decimal precision overflow has no wrap-around semantics, so silent NULL or corruption is the only alternative to failing. Compatibility consequence, the same one #7906/#7992 accepted for integral types: jobs that today "succeed" while writing corrupt decimals will now fail with `ArithmeticException`. ### Tests - `DecimalTest#testAddSubtractPrecisionOverflow` (paimon-common): compact and non-compact `add`/`subtract` return null instead of an out-of-range `Decimal`, for both `DECIMAL(5,0)` and `DECIMAL(5,2)` (the bound is `10^precision`, not `10^(precision+scale)`). Fails without the fix: `expected: null but was: 100000`. - `FieldAggregatorTest#testFieldSumDecimalOverflow` / `#testFieldSumDecimalRetractOverflow` (compact, `DECIMAL(5,0)`), `#testFieldSumDecimalOverflowWithScale` (compact, `DECIMAL(5,2)`) and `#testFieldSumDecimalOverflowNonCompact` (`DECIMAL(38,2)`): `agg`/`retract` throw `ArithmeticException` on overflow. All fail without the fix. - Existing `DecimalTest` cases exercising `add()` and the full `FieldAggregatorTest` pass unchanged. ### API and Format No. ### Documentation No. The `fail-on-overflow` docs added in #9874 cover integral types only; a decimal precision overflow has no ignore mode to document. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
