dongjoon-hyun opened a new pull request, #60188:
URL: https://github.com/apache/spark/pull/60188
### What changes were proposed in this pull request?
This PR fixes decimal `%` (`mod`, `try_mod`), `pmod` and `div` raising an
unclassified `java.lang.ArithmeticException: Division impossible` when the
integral part of the quotient has more than 39 digits, e.g. for `DECIMAL(38,
0)` and `DECIMAL(3, 3)` operands.
`Decimal.%` and `Decimal.quot` passed `Decimal.MATH_CONTEXT` (precision 39,
`RoundingMode.DOWN`) to `java.math.BigDecimal.remainder` and
`divideToIntegralValue`. These methods raise `Division impossible` when the
integral part of the exact quotient needs more digits than the `MathContext`
holds. For two decimals of precision up to 38, it can need up to 76 digits.
This PR passes a `MathContext` that holds the integral part instead. `|a| <
10^(a.precision - a.scale)` and `|b| >= 10^(b.precision - b.scale - 1)`, so the
integral part of `a / b` has at most `(a.precision - a.scale) - (b.precision -
b.scale) + 1` digits. When this bound is at most 39, `MATH_CONTEXT` is used as
before. Otherwise, a `MathContext` of that precision is used.
- `%`, `try_mod` and `pmod` now return the exact remainder in every mode. It
always fits the result type, because it is smaller than the divisor and its
scale is at most `max(s1, s2)`.
- `div` now passes the exact quotient to the existing overflow check of
`IntegralDivide`. It returns `NULL` in non-ANSI mode and raises
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, which `div` already does when the
quotient has 39 digits.
The interpreted and the code-generated paths both call these two methods, so
both are fixed. Calls that did not raise before return the same values, because
`divideToIntegralValue` returns the exact integral part whenever it does not
raise.
### Why are the changes needed?
Spark returns `NULL` for a decimal overflow in non-ANSI mode and in `try_*`
functions, and raises a Spark error class in ANSI mode. These operators raise a
raw `ArithmeticException` instead, even in non-ANSI mode and in `try_mod`,
although the remainder is representable. Whether it raises depends on the
values, not only on the types. With the same dividend, `0.00001` (an exact
43-digit quotient) and `0.7` (a 39-digit quotient) work, while `0.007` (a
41-digit quotient) raises.
```sql
-- With spark.sql.ansi.enabled=false, each query should return 0.006 or
NULL, but raises
-- java.lang.ArithmeticException: Division impossible
SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) % 0.007BD;
SELECT try_mod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
SELECT pmod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) div 0.007BD;
```
The same happens in ANSI mode, and when the operands are columns, where the
task fails with the same exception.
The limit was 38 digits before SPARK-45786 (Apache Spark 3.5.1 and 4.0.0)
and has been 39 digits since then.
### Does this PR introduce _any_ user-facing change?
Yes. For the inputs above, `%`, `try_mod` and `pmod` return `0.006` instead
of raising. `div` returns `NULL` in non-ANSI mode and raises
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, instead of raising
`ArithmeticException: Division impossible`. Other results do not change.
### How was this patch tested?
Pass the CIs with a newly added test case in `ArithmeticExpressionSuite`. It
checks the interpreted and the code-generated paths.
I also compared the old and the new `MathContext` with a local
micro-benchmark on random operands, and the new one costs the same when the
bound is at most 39 digits. Dropping the `MathContext`, which also gives the
exact quotient, was 1.3x to 2.7x slower there.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
