viirya commented on code in PR #59334:
URL: https://github.com/apache/spark/pull/59334#discussion_r4251417666
##########
sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala:
##########
@@ -648,8 +649,10 @@ object Decimal {
try {
val bigDecimal = stringToJavaBigDecimal(str)
// We fast fail because constructing a very large JavaBigDecimal to
Decimal is very slow.
- // For example: Decimal("6.0790316E+25569151")
- if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
+ // For example: Decimal("6.0790316E+25569151"). Zero fits any decimal
type whatever its
+ // exponent, e.g. "0e39", so it is not checked.
+ if (bigDecimal.signum != 0 &&
Review Comment:
Done. When negative scales are not allowed, a zero's negative scale is
dropped before `Decimal.set`, so the result has precision 1 and scale 0;
`'0.00'` keeps scale 2.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala:
##########
@@ -446,6 +446,24 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester with SQLHelper
}
}
+ test("UTF8String to Decimal with exponent near Int.MaxValue") {
+ // The number of integral digits (precision - scale) of these values
overflows Int.
+ Seq("1e2147483647", "-1E+2147483647", "12e2147483647",
+ "1.5e2147483647").foreach { string =>
+ assert(Decimal.fromString(UTF8String.fromString(string)) === null)
+ checkError(
+ exception = intercept[SparkArithmeticException](
+ Decimal.fromStringANSI(UTF8String.fromString(string))),
+ condition = "NUMERIC_OUT_OF_SUPPORTED_RANGE",
+ parameters = Map("value" -> string))
+ }
+ // Zero fits any decimal type whatever its exponent.
+ Seq("0e39", "-0E+100", "0e2147483647").foreach { string =>
+ assert(Decimal.fromString(UTF8String.fromString(string)) === Decimal(0))
Review Comment:
Done. The test now also asserts precision 1 and scale 0 for these zeros, and
fails with "40 did not equal 1" without the normalization.
##########
sql/api/src/main/scala/org/apache/spark/sql/types/Decimal.scala:
##########
@@ -668,8 +671,10 @@ object Decimal {
try {
val bigDecimal = stringToJavaBigDecimal(str)
// We fast fail because constructing a very large JavaBigDecimal to
Decimal is very slow.
- // For example: Decimal("6.0790316E+25569151")
- if (numDigitsInIntegralPart(bigDecimal) > DecimalType.MAX_PRECISION &&
+ // For example: Decimal("6.0790316E+25569151"). Zero fits any decimal
type whatever its
+ // exponent, e.g. "0e39", so it is not checked.
+ if (bigDecimal.signum != 0 &&
Review Comment:
Done. Moved into one private helper, `stringBigDecimalToDecimal`, which also
holds the zero rule.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/CastWithAnsiOnSuite.scala:
##########
@@ -475,6 +475,14 @@ class CastWithAnsiOnSuite extends CastSuiteBase with
QueryErrorsBase {
checkExceptionInExpression[ArithmeticException](
cast("6E+37", DecimalType(38, 1)),
"cannot be represented as Decimal(38, 1)")
+ Seq("1e2147483647", "1.5e2147483647", "12e2147483647").foreach { s =>
+ checkExceptionInExpression[ArithmeticException](
+ cast(s, DecimalType(10, 2)),
+ "NUMERIC_OUT_OF_SUPPORTED_RANGE")
+ }
+ Seq("0e39", "0e2147483647").foreach { s =>
Review Comment:
Done. Moved to `CastSuiteBase`, so they run for ANSI on, ANSI off and
`try_cast`.
##########
sql/catalyst/src/test/scala/org/apache/spark/sql/types/DecimalSuite.scala:
##########
@@ -446,6 +446,24 @@ class DecimalSuite extends SparkFunSuite with
PrivateMethodTester with SQLHelper
}
}
+ test("UTF8String to Decimal with exponent near Int.MaxValue") {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
