sundapeng commented on code in PR #11308:
URL: https://github.com/apache/paimon/pull/11308#discussion_r4657491024
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldSumAgg.java:
##########
@@ -106,6 +115,15 @@ public Object retract(Object accumulator, Object
inputField) {
inFieldDD,
mergeFieldDD.precision(),
mergeFieldDD.scale());
+ if (sum == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Decimal overflow: %s - %s (precision=%d,
scale=%d)",
+ mergeFieldDD.toBigDecimal(),
+ inFieldDD.toBigDecimal(),
+ mergeFieldDD.precision(),
+ mergeFieldDD.scale()));
+ }
Review Comment:
Good point — the unconditional throw ignored
`fields.<field>.sum.fail-on-overflow` and could indeed put compaction /
merge-on-read over committed files into an unrecoverable failure loop.
35e753d1b7 makes the DECIMAL branch honor the option, mirroring the integral
types:
- `fail-on-overflow = true` → throw `ArithmeticException` (as before);
- `fail-on-overflow = false` (the default) → the aggregated value becomes
NULL, the same result `FieldProductAgg` already produces on a DECIMAL overflow,
and the same "overflows are ignored by default" contract as
TINYINT/SMALLINT/INTEGER/BIGINT.
The default is still a strict improvement over master: with the
`DecimalUtils` bounds check, both the compact and the non-compact path now
return NULL, whereas the compact path used to return a `Decimal` whose unscaled
value violated the declared precision and was written to the data files.
Docs are updated in the same commit (`aggregation.mdx`: a DECIMAL precision
overflow makes the aggregated value NULL by default), and
`testFieldSumDecimalOverflowDisabled` pins the default mode for compact and
non-compact, agg and retract.
##########
paimon-common/src/main/java/org/apache/paimon/utils/DecimalUtils.java:
##########
@@ -77,7 +79,9 @@ public static Decimal subtract(Decimal v1, Decimal v2, int
precision, int scale)
long ls =
Math.subtractExact(
v1.toUnscaledLong(), v2.toUnscaledLong()); //
checks overflow
- return fromUnscaledLong(ls, precision, scale);
+ if (ls < POW10[precision] && ls > -POW10[precision]) {
+ return fromUnscaledLong(ls, precision, scale);
Review Comment:
Applied in 35e753d1b7, with one refinement: the fast path now also requires
`scale == v1.scale()` before returning null directly. Without that, a call
whose target scale differs from the operands' scale — previously guarded only
by `assert scale == v1.scale()`, i.e. silently wrong in production with asserts
disabled — would return null instead of rescaling through `fromBigDecimal`.
That case is now pinned by new assertions in
`DecimalTest#testAddSubtractPrecisionOverflow` (`add(10, 15, 5, 2)` must be
`25.00`, not `0.25`).
With the condition in place, `ls` is the exact unscaled result at the target
scale, so on a bound miss or an `addExact`/`subtractExact` overflow the
BigDecimal path is guaranteed to return null (`|ls| >= 10^precision` implies
`bd.precision() > precision`; a long overflow implies `|sum| >= 2^63 > 10^18 >=
10^precision`) — returning null directly skips the three BigDecimal allocations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]