Akash3121 commented on code in PR #11308:
URL: https://github.com/apache/paimon/pull/11308#discussion_r4648961771
##########
paimon-core/src/main/java/org/apache/paimon/mergetree/compact/aggregate/FieldSumAgg.java:
##########
@@ -106,6 +115,15 @@ public Object retract(Object accumulator, Object
inputField) {
inFieldDD,
mergeFieldDD.precision(),
mergeFieldDD.scale());
+ if (sum == null) {
+ throw new ArithmeticException(
+ String.format(
+ "Decimal overflow: %s - %s (precision=%d,
scale=%d)",
+ mergeFieldDD.toBigDecimal(),
+ inFieldDD.toBigDecimal(),
+ mergeFieldDD.precision(),
+ mergeFieldDD.scale()));
+ }
Review Comment:
In #9874, `fields.<field-name>.sum.fail-on-overflow` (default `false`) was
introduced after #7906 and #7992 because `FieldSumAgg` executes during LSM
compaction and merge-on-read over already-committed Level-0 SST files. If
individual batches write valid `DECIMAL(p, s)` values that only overflow when
merged across sorted runs during compaction, throwing an unconditional
`ArithmeticException` (ignoring `failOnOverflow`) can put compaction and
readers into an unrecoverable failure loop with no table option to bypass it.
Also note that before this change, non-compact `DECIMAL` (`precision > 18`)
returned `null` on overflow in `FieldSumAgg` (and `FieldProductAgg` still
returns `null` on `DECIMAL` overflow).
1. Should this check respect `failOnOverflow` (i.e. `if (failOnOverflow &&
sum == null)`), or is there a fallback strategy when `failOnOverflow == false`?
2. If `DECIMAL` is intentionally meant to always throw on overflow
regardless of `fields.<field-name>.sum.fail-on-overflow`, could we also update
`docs/docs/primary-key-table/merge-engine/aggregation.mdx` under `### sum` to
clarify that `DECIMAL` always throws `ArithmeticException` on precision
overflow?
##########
paimon-common/src/main/java/org/apache/paimon/utils/DecimalUtils.java:
##########
@@ -77,7 +79,9 @@ public static Decimal subtract(Decimal v1, Decimal v2, int
precision, int scale)
long ls =
Math.subtractExact(
v1.toUnscaledLong(), v2.toUnscaledLong()); //
checks overflow
- return fromUnscaledLong(ls, precision, scale);
+ if (ls < POW10[precision] && ls > -POW10[precision]) {
+ return fromUnscaledLong(ls, precision, scale);
Review Comment:
In this compact branch (`v1.isCompact() && v2.isCompact() && v1.scale() ==
v2.scale() && Decimal.isCompact(precision)` with `scale == v1.scale()`), `ls`
is already the exact unscaled result at the target `scale`. When `|ls| >=
POW10[precision]` (or `Math.addExact` / `Math.subtractExact` throws
`ArithmeticException`), falling through to
`v1.toBigDecimal().add(v2.toBigDecimal())` and `fromBigDecimal(bd, precision,
scale)` is guaranteed to have `bd.precision() > precision` and return `null`,
after unnecessarily allocating three `BigDecimal` objects.
We can return `null` directly from the compact branch on overflow (and
similarly in `subtract` below):
```java
if (ls < POW10[precision] && ls > -POW10[precision]) {
return Decimal.fromUnscaledLong(ls, precision, scale);
}
} catch (ArithmeticException ignored) {
}
return null;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]