This is an automated email from the ASF dual-hosted git repository.
dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.3 by this push:
new d94c350ce737 [SPARK-61050][SQL] Fix `ArithmeticException` of decimal
remainder, `pmod` and `div` when the integral quotient has more than 39 digits
d94c350ce737 is described below
commit d94c350ce737cd5dca33cad03a50adac6bb9282b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 7 09:21:11 2026 -0800
[SPARK-61050][SQL] Fix `ArithmeticException` of decimal remainder, `pmod`
and `div` when the integral quotient has more than 39 digits
### What changes were proposed in this pull request?
This PR fixes decimal `%` (`mod`, `try_mod`), `pmod` and `div` raising an
unclassified `java.lang.ArithmeticException: Division impossible` when the
integral part of the quotient has more than 39 digits, e.g. for `DECIMAL(38,
0)` and `DECIMAL(3, 3)` operands.
`Decimal.%` and `Decimal.quot` passed `Decimal.MATH_CONTEXT` (precision 39,
`RoundingMode.DOWN`) to `java.math.BigDecimal.remainder` and
`divideToIntegralValue`. These methods raise `Division impossible` when the
integral part of the exact quotient needs more digits than the `MathContext`
holds. For two decimals of precision up to 38, it can need up to 76 digits.
This PR still tries `MATH_CONTEXT` first. Only when it raises, the call is
retried with a `MathContext` that holds the integral part. `|a| <
10^(a.precision - a.scale)` and `|b| >= 10^(b.precision - b.scale - 1)`, so the
integral part of `a / b` has at most `(a.precision - a.scale) - (b.precision -
b.scale) + 1` digits. When this bound is at most 39 or beyond the `Int` range,
the original exception is rethrown.
A wider `MathContext` is not used up front, because it computes more digits
even when `MATH_CONTEXT` already holds the quotient. This makes exact quotients
such as `CAST(repeat('9', 38) AS DECIMAL(38, 0)) % 1E-38BD` about 9x slower.
With the negative scales of `spark.sql.legacy.allowNegativeScaleOfDecimal`, it
makes `1E+100000BD % 1BD` take seconds and `1E+999999999BD % 1BD` overflow
`BigInteger`.
- `%`, `try_mod` and `pmod` now return the exact remainder in every mode.
It always fits the result type, because it is smaller than the divisor and its
scale is at most `max(s1, s2)`.
- `div` now passes the exact quotient to the existing overflow check of
`IntegralDivide`. It returns `NULL` in non-ANSI mode and raises
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, which `div` already does when the
quotient has 39 digits.
The interpreted and the code-generated paths both call these two methods,
so both are fixed. Calls that did not raise before make the same call as
before, so their values, scales and costs do not change.
### Why are the changes needed?
Spark returns `NULL` for a decimal overflow in non-ANSI mode and in `try_*`
functions, and raises a Spark error class in ANSI mode. These operators raise a
raw `ArithmeticException` instead, even in non-ANSI mode and in `try_mod`,
although the remainder is representable. Whether it raises depends on the
values, not only on the types. With the same dividend, `0.00001` (an exact
43-digit quotient) and `0.7` (a 39-digit quotient) work, while `0.007` (a
41-digit quotient) raises.
```sql
-- With spark.sql.ansi.enabled=false, each query should return 0.006 or
NULL, but raises
-- java.lang.ArithmeticException: Division impossible
SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) % 0.007BD;
SELECT try_mod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
SELECT pmod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) div 0.007BD;
```
The same happens in ANSI mode, and when the operands are columns, where the
task fails with the same exception.
The limit was 38 digits before SPARK-45786 (Apache Spark 3.5.1 and 4.0.0)
and has been 39 digits since then.
### Does this PR introduce _any_ user-facing change?
Yes. For the inputs above, `%`, `try_mod` and `pmod` return `0.006` instead
of raising. `div` returns `NULL` in non-ANSI mode and raises
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, instead of raising
`ArithmeticException: Division impossible`. Other results do not change.
### How was this patch tested?
Pass the CIs with newly added test cases.
- `ArithmeticExpressionSuite` checks 40- and 76-digit quotients on the
interpreted and the code-generated paths.
- `DecimalSuite` checks `%` and `quot` with negative scales.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
Closes #60188 from dongjoon-hyun/SPARK-61050.
Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>
(cherry picked from commit 00ce94fdba0d4800c3c631583436623c127415c9)
Signed-off-by: Dongjoon Hyun <[email protected]>
---
.../scala/org/apache/spark/sql/types/Decimal.scala | 41 ++++++++++++++++++++--
.../expressions/ArithmeticExpressionSuite.scala | 14 ++++++++
.../org/apache/spark/sql/types/DecimalSuite.scala | 13 +++++++
3 files changed, 66 insertions(+), 2 deletions(-)
diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
index 1504b55b3af2..5c35460f1f67 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
@@ -556,14 +556,51 @@ final class Decimal extends Ordered[Decimal] with
Serializable {
def %(that: Decimal): Decimal =
if (that.isZero) null
- else Decimal(toJavaBigDecimal.remainder(that.toJavaBigDecimal,
MATH_CONTEXT))
+ else {
+ val (a, b) = (toJavaBigDecimal, that.toJavaBigDecimal)
+ val r =
+ try a.remainder(b, MATH_CONTEXT)
+ catch {
+ case e: ArithmeticException =>
+ a.remainder(b, integralQuotientContext(a, b, e))
+ }
+ Decimal(r)
+ }
def quot(that: Decimal): Decimal =
if (that.isZero) null
- else Decimal(toJavaBigDecimal.divideToIntegralValue(that.toJavaBigDecimal,
MATH_CONTEXT))
+ else {
+ val (a, b) = (toJavaBigDecimal, that.toJavaBigDecimal)
+ val q =
+ try a.divideToIntegralValue(b, MATH_CONTEXT)
+ catch {
+ case e: ArithmeticException =>
+ a.divideToIntegralValue(b, integralQuotientContext(a, b, e))
+ }
+ Decimal(q)
+ }
def remainder(that: Decimal): Decimal = this % that
+ // With `MATH_CONTEXT`, `BigDecimal.remainder` and `divideToIntegralValue`
raise "Division
+ // impossible" when the integral part of the quotient has more than 39
digits, e.g. 41 digits
+ // for DECIMAL(38, 0) and DECIMAL(3, 3). `%` and `quot` then retry with the
context returned
+ // here, which holds the integral part of `a / b`: |a| < 10^(a.precision -
a.scale) and
+ // |b| >= 10^(b.precision - b.scale - 1). They try `MATH_CONTEXT` first,
because a wider
+ // context computes more digits, which can be very many with negative
scales. `e` is rethrown
+ // when the bound is within `MATH_CONTEXT` or beyond the Int range.
+ // This is a private method of the class, not of `object Decimal`: the class
calling a private
+ // member of the object makes it public in `Decimal$` and changes the
computed
+ // serialVersionUID of `Decimal$`, which
`UdfSerialization.auditedTransitions` pins.
+ private def integralQuotientContext(
+ a: JavaBigDecimal,
+ b: JavaBigDecimal,
+ e: ArithmeticException): MathContext = {
+ val digits = (a.precision.toLong - a.scale) - (b.precision.toLong -
b.scale) + 1
+ if (digits <= MATH_CONTEXT.getPrecision || digits > Int.MaxValue) throw e
+ new MathContext(digits.toInt, RoundingMode.DOWN)
+ }
+
def unary_- : Decimal = {
if (decimalVal.ne(null)) {
Decimal(-decimalVal, precision, scale)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
index f8cdf825bc4c..ceb5e96971db 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
@@ -756,6 +756,20 @@ class ArithmeticExpressionSuite extends SparkFunSuite with
ExpressionEvalHelper
}
}
+ test("SPARK-61050: decimal Remainder, Pmod and IntegralDivide with a 40- or
76-digit quotient") {
+ val left = Literal(Decimal(BigDecimal("9" * 38)), DecimalType(38, 0))
+ Seq(("0.07", 2, "0.02"), ("7E-38", 38, "2E-38")).foreach { case (divisor,
scale, rem) =>
+ val right = Literal(Decimal(BigDecimal(divisor), scale, scale),
DecimalType(scale, scale))
+ Seq(EvalMode.LEGACY, EvalMode.ANSI).foreach { mode =>
+ checkEvaluation(Remainder(left, right, mode), Decimal(BigDecimal(rem),
scale, scale))
+ checkEvaluation(Pmod(left, right, mode), Decimal(BigDecimal(rem),
scale, scale))
+ }
+ checkEvaluation(IntegralDivide(left, right, EvalMode.LEGACY), null)
+ checkExceptionInExpression[SparkArithmeticException](
+ IntegralDivide(left, right, EvalMode.ANSI),
"NUMERIC_VALUE_OUT_OF_RANGE")
+ }
+ }
+
test("SPARK-24598: overflow on long returns wrong result") {
val maxLongLiteral = Literal(Long.MaxValue)
val minLongLiteral = Literal(Long.MinValue)
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
index f6cd34df8898..105d419172f7 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
@@ -258,6 +258,19 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester with SQLHelper
assert(Decimal(100).quot(Decimal(-33)) === Decimal(BigDecimal("-3")))
}
+ test("SPARK-61050: % and quot with negative scales") {
+ withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key ->
"true") {
+ // `MATH_CONTEXT` holds this exact quotient, so no context of 10^9
digits is used.
+ val big = Decimal(BigDecimal("1E+999999999"))
+ assert(big % Decimal(1) === Decimal(0))
+ assert(big.quot(Decimal(1)) === big)
+ // The integral part of the quotient would need more digits than an Int
can count.
+ intercept[ArithmeticException] {
+ Decimal(BigDecimal("1E+2147483636")) % Decimal(BigDecimal("3E-38"))
+ }
+ }
+ }
+
test("SPARK-57937: div with target scale") {
// `div(that, scale)` must return the same result as `/` (which divides at
// `DecimalType.MAX_SCALE + 1` with ROUND_DOWN) followed by rounding to
the same scale.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]