This is an automated email from the ASF dual-hosted git repository.

dongjoon-hyun pushed a commit to branch branch-4.3
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/branch-4.3 by this push:
     new d94c350ce737 [SPARK-61050][SQL] Fix `ArithmeticException` of decimal 
remainder, `pmod` and `div` when the integral quotient has more than 39 digits
d94c350ce737 is described below

commit d94c350ce737cd5dca33cad03a50adac6bb9282b
Author: Dongjoon Hyun <[email protected]>
AuthorDate: Sat Nov 7 09:21:11 2026 -0800

    [SPARK-61050][SQL] Fix `ArithmeticException` of decimal remainder, `pmod` 
and `div` when the integral quotient has more than 39 digits
    
    ### What changes were proposed in this pull request?
    
    This PR fixes decimal `%` (`mod`, `try_mod`), `pmod` and `div` raising an 
unclassified `java.lang.ArithmeticException: Division impossible` when the 
integral part of the quotient has more than 39 digits, e.g. for `DECIMAL(38, 
0)` and `DECIMAL(3, 3)` operands.
    
    `Decimal.%` and `Decimal.quot` passed `Decimal.MATH_CONTEXT` (precision 39, 
`RoundingMode.DOWN`) to `java.math.BigDecimal.remainder` and 
`divideToIntegralValue`. These methods raise `Division impossible` when the 
integral part of the exact quotient needs more digits than the `MathContext` 
holds. For two decimals of precision up to 38, it can need up to 76 digits.
    
    This PR still tries `MATH_CONTEXT` first. Only when it raises, the call is 
retried with a `MathContext` that holds the integral part. `|a| < 
10^(a.precision - a.scale)` and `|b| >= 10^(b.precision - b.scale - 1)`, so the 
integral part of `a / b` has at most `(a.precision - a.scale) - (b.precision - 
b.scale) + 1` digits. When this bound is at most 39 or beyond the `Int` range, 
the original exception is rethrown.
    
    A wider `MathContext` is not used up front, because it computes more digits 
even when `MATH_CONTEXT` already holds the quotient. This makes exact quotients 
such as `CAST(repeat('9', 38) AS DECIMAL(38, 0)) % 1E-38BD` about 9x slower. 
With the negative scales of `spark.sql.legacy.allowNegativeScaleOfDecimal`, it 
makes `1E+100000BD % 1BD` take seconds and `1E+999999999BD % 1BD` overflow 
`BigInteger`.
    
    - `%`, `try_mod` and `pmod` now return the exact remainder in every mode. 
It always fits the result type, because it is smaller than the divisor and its 
scale is at most `max(s1, s2)`.
    - `div` now passes the exact quotient to the existing overflow check of 
`IntegralDivide`. It returns `NULL` in non-ANSI mode and raises 
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, which `div` already does when the 
quotient has 39 digits.
    
    The interpreted and the code-generated paths both call these two methods, 
so both are fixed. Calls that did not raise before make the same call as 
before, so their values, scales and costs do not change.
    
    ### Why are the changes needed?
    
    Spark returns `NULL` for a decimal overflow in non-ANSI mode and in `try_*` 
functions, and raises a Spark error class in ANSI mode. These operators raise a 
raw `ArithmeticException` instead, even in non-ANSI mode and in `try_mod`, 
although the remainder is representable. Whether it raises depends on the 
values, not only on the types. With the same dividend, `0.00001` (an exact 
43-digit quotient) and `0.7` (a 39-digit quotient) work, while `0.007` (a 
41-digit quotient) raises.
    
    ```sql
    -- With spark.sql.ansi.enabled=false, each query should return 0.006 or 
NULL, but raises
    -- java.lang.ArithmeticException: Division impossible
    SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) % 0.007BD;
    SELECT try_mod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
    SELECT pmod(CAST(repeat('9', 38) AS DECIMAL(38, 0)), 0.007BD);
    SELECT CAST(repeat('9', 38) AS DECIMAL(38, 0)) div 0.007BD;
    ```
    
    The same happens in ANSI mode, and when the operands are columns, where the 
task fails with the same exception.
    
    The limit was 38 digits before SPARK-45786 (Apache Spark 3.5.1 and 4.0.0) 
and has been 39 digits since then.
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. For the inputs above, `%`, `try_mod` and `pmod` return `0.006` instead 
of raising. `div` returns `NULL` in non-ANSI mode and raises 
`NUMERIC_VALUE_OUT_OF_RANGE` in ANSI mode, instead of raising 
`ArithmeticException: Division impossible`. Other results do not change.
    
    ### How was this patch tested?
    
    Pass the CIs with newly added test cases.
    - `ArithmeticExpressionSuite` checks 40- and 76-digit quotients on the 
interpreted and the code-generated paths.
    - `DecimalSuite` checks `%` and `quot` with negative scales.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: Claude Opus 5.5
    
    Closes #60188 from dongjoon-hyun/SPARK-61050.
    
    Authored-by: Dongjoon Hyun <[email protected]>
    Signed-off-by: Dongjoon Hyun <[email protected]>
    
    (cherry picked from commit 00ce94fdba0d4800c3c631583436623c127415c9)
    Signed-off-by: Dongjoon Hyun <[email protected]>
---
 .../scala/org/apache/spark/sql/types/Decimal.scala | 41 ++++++++++++++++++++--
 .../expressions/ArithmeticExpressionSuite.scala    | 14 ++++++++
 .../org/apache/spark/sql/types/DecimalSuite.scala  | 13 +++++++
 3 files changed, 66 insertions(+), 2 deletions(-)

diff --git a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala 
b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
index 1504b55b3af2..5c35460f1f67 100644
--- a/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
+++ b/sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala
@@ -556,14 +556,51 @@ final class Decimal extends Ordered[Decimal] with 
Serializable {
 
   def %(that: Decimal): Decimal =
     if (that.isZero) null
-    else Decimal(toJavaBigDecimal.remainder(that.toJavaBigDecimal, 
MATH_CONTEXT))
+    else {
+      val (a, b) = (toJavaBigDecimal, that.toJavaBigDecimal)
+      val r =
+        try a.remainder(b, MATH_CONTEXT)
+        catch {
+          case e: ArithmeticException =>
+            a.remainder(b, integralQuotientContext(a, b, e))
+        }
+      Decimal(r)
+    }
 
   def quot(that: Decimal): Decimal =
     if (that.isZero) null
-    else Decimal(toJavaBigDecimal.divideToIntegralValue(that.toJavaBigDecimal, 
MATH_CONTEXT))
+    else {
+      val (a, b) = (toJavaBigDecimal, that.toJavaBigDecimal)
+      val q =
+        try a.divideToIntegralValue(b, MATH_CONTEXT)
+        catch {
+          case e: ArithmeticException =>
+            a.divideToIntegralValue(b, integralQuotientContext(a, b, e))
+        }
+      Decimal(q)
+    }
 
   def remainder(that: Decimal): Decimal = this % that
 
+  // With `MATH_CONTEXT`, `BigDecimal.remainder` and `divideToIntegralValue` 
raise "Division
+  // impossible" when the integral part of the quotient has more than 39 
digits, e.g. 41 digits
+  // for DECIMAL(38, 0) and DECIMAL(3, 3). `%` and `quot` then retry with the 
context returned
+  // here, which holds the integral part of `a / b`: |a| < 10^(a.precision - 
a.scale) and
+  // |b| >= 10^(b.precision - b.scale - 1). They try `MATH_CONTEXT` first, 
because a wider
+  // context computes more digits, which can be very many with negative 
scales. `e` is rethrown
+  // when the bound is within `MATH_CONTEXT` or beyond the Int range.
+  // This is a private method of the class, not of `object Decimal`: the class 
calling a private
+  // member of the object makes it public in `Decimal$` and changes the 
computed
+  // serialVersionUID of `Decimal$`, which 
`UdfSerialization.auditedTransitions` pins.
+  private def integralQuotientContext(
+      a: JavaBigDecimal,
+      b: JavaBigDecimal,
+      e: ArithmeticException): MathContext = {
+    val digits = (a.precision.toLong - a.scale) - (b.precision.toLong - 
b.scale) + 1
+    if (digits <= MATH_CONTEXT.getPrecision || digits > Int.MaxValue) throw e
+    new MathContext(digits.toInt, RoundingMode.DOWN)
+  }
+
   def unary_- : Decimal = {
     if (decimalVal.ne(null)) {
       Decimal(-decimalVal, precision, scale)
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
index f8cdf825bc4c..ceb5e96971db 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/ArithmeticExpressionSuite.scala
@@ -756,6 +756,20 @@ class ArithmeticExpressionSuite extends SparkFunSuite with 
ExpressionEvalHelper
     }
   }
 
+  test("SPARK-61050: decimal Remainder, Pmod and IntegralDivide with a 40- or 
76-digit quotient") {
+    val left = Literal(Decimal(BigDecimal("9" * 38)), DecimalType(38, 0))
+    Seq(("0.07", 2, "0.02"), ("7E-38", 38, "2E-38")).foreach { case (divisor, 
scale, rem) =>
+      val right = Literal(Decimal(BigDecimal(divisor), scale, scale), 
DecimalType(scale, scale))
+      Seq(EvalMode.LEGACY, EvalMode.ANSI).foreach { mode =>
+        checkEvaluation(Remainder(left, right, mode), Decimal(BigDecimal(rem), 
scale, scale))
+        checkEvaluation(Pmod(left, right, mode), Decimal(BigDecimal(rem), 
scale, scale))
+      }
+      checkEvaluation(IntegralDivide(left, right, EvalMode.LEGACY), null)
+      checkExceptionInExpression[SparkArithmeticException](
+        IntegralDivide(left, right, EvalMode.ANSI), 
"NUMERIC_VALUE_OUT_OF_RANGE")
+    }
+  }
+
   test("SPARK-24598: overflow on long returns wrong result") {
     val maxLongLiteral = Literal(Long.MaxValue)
     val minLongLiteral = Literal(Long.MinValue)
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
index f6cd34df8898..105d419172f7 100644
--- a/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
+++ b/sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala
@@ -258,6 +258,19 @@ class DecimalSuite extends SparkFunSuite with 
PrivateMethodTester with SQLHelper
     assert(Decimal(100).quot(Decimal(-33)) === Decimal(BigDecimal("-3")))
   }
 
+  test("SPARK-61050: % and quot with negative scales") {
+    withSQLConf(SQLConf.LEGACY_ALLOW_NEGATIVE_SCALE_OF_DECIMAL_ENABLED.key -> 
"true") {
+      // `MATH_CONTEXT` holds this exact quotient, so no context of 10^9 
digits is used.
+      val big = Decimal(BigDecimal("1E+999999999"))
+      assert(big % Decimal(1) === Decimal(0))
+      assert(big.quot(Decimal(1)) === big)
+      // The integral part of the quotient would need more digits than an Int 
can count.
+      intercept[ArithmeticException] {
+        Decimal(BigDecimal("1E+2147483636")) % Decimal(BigDecimal("3E-38"))
+      }
+    }
+  }
+
   test("SPARK-57937: div with target scale") {
     // `div(that, scale)` must return the same result as `/` (which divides at
     // `DecimalType.MAX_SCALE + 1` with ROUND_DOWN) followed by rounding to 
the same scale.


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to