HyukjinKwon commented on code in PR #59334:
URL: https://github.com/apache/spark/pull/59334#discussion_r4242296137
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala:
##########
@@ -446,6 +446,24 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester with SQLHelper
}
}
+ test("UTF8String to Decimal with exponent near Int.MaxValue") {
+ // The number of integral digits (precision - scale) of these values
overflows Int.
+ Seq("1e2147483647", "-1E+2147483647", "12e2147483647", "1.5e2147483647",
+ "1e2147483648").foreach { string =>
Review Comment:
**Blocking (P1):** `"1e2147483648"` (and `"0.0e2147483648"` at line 461)
only parse on newer JDKs. On JDK 17, which is Spark's `java.version` and the
default CI JDK, `new java.math.BigDecimal(s)` throws
`NumberFormatException("Exponent overflow.")` for an exponent outside `Int`
range. That happens in `stringToJavaBigDecimal`, before the fast-fail runs. So
on JDK 17:
- `fromStringANSI("1e2147483648")` throws `SparkNumberFormatException`
(`CAST_INVALID_INPUT`), not `SparkArithmeticException` /
`NUMERIC_OUT_OF_SUPPORTED_RANGE`.
- `"0.0e2147483648"` gives null / `CAST_INVALID_INPUT` instead of zero.
JDK 25 parses both strings (scale -2147483648 and -2147483647). I checked
both JDKs with a standalone `BigDecimal` probe.
This is what fails CI on this commit:
- `DecimalSuite`: this test
- `CastWithAnsiOnSuite`: lines 478 and 483, and `TryCastSuite`, which
inherits them
- `CastWithAnsiOffSuite`: line 565
The `"1e2147483648"` -> null checks still pass on JDK 17, but only through
the `NumberFormatException` path, not the `Long` fast-fail they are meant to
cover.
Could you drop these two inputs, or replace them with strings that parse on
every supported JDK and still overflow `Int`? For example, `"10e2147483647"`
(precision 2, scale -2147483647) for the non-zero side and `"0.0e2147483647"`
for the zero side.
**Verification:**
- **Regression:** On JDK 17, the new Decimal string-conversion test and the
cast-string-to-decimal fast-fail tests (ANSI on, ANSI off, and try_cast,
interpreted and codegen) pass, where they currently fail on the
exponent-2147483648 inputs.
- **Behavior:** Each non-zero test input parses with java.math.BigDecimal on
JDK 17 and yields precision - scale greater than Int.MaxValue, and the same
assertions fail if numDigitsInIntegralPart is reverted to Int arithmetic.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
