[
https://issues.apache.org/jira/browse/SPARK-60118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh resolved SPARK-60118.
---------------------------------
Fix Version/s: 4.2.2
4.1.5
4.0.6
4.3.1
4.4.0
Resolution: Fixed
Issue resolved by pull request 59334
[https://github.com/apache/spark/pull/59334]
> Casting a string with an exponent near Int.MaxValue to decimal throws
> ArithmeticException instead of NULL / NUMERIC_OUT_OF_SUPPORTED_RANGE
> ------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60118
> URL: https://issues.apache.org/jira/browse/SPARK-60118
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.2, 4.1.5, 4.0.6, 4.3.1, 4.4.0
>
>
> Casting a string whose exponent is near Int.MaxValue to a decimal type fails
> with a raw java.lang.ArithmeticException under all cast modes, instead of
> returning NULL (try_cast / legacy cast) or raising
> NUMERIC_OUT_OF_SUPPORTED_RANGE (ANSI cast):
> {code:sql}
> SELECT try_cast('1e2147483647' AS DECIMAL(10,2));
> {code}
> {code}
> java.lang.ArithmeticException: BigInteger would overflow supported range
> at java.base/java.math.BigInteger.reportOverflow(BigInteger.java:1178)
> at java.base/java.math.BigInteger.pow(BigInteger.java:2627)
> at java.base/java.math.BigDecimal.bigTenToThe(BigDecimal.java:4375)
> at java.base/java.math.BigDecimal.bigMultiplyPowerTen(BigDecimal.java:4510)
> at java.base/java.math.BigDecimal.setScale(BigDecimal.java:2936)
> at java.base/java.math.BigDecimal.setScale(BigDecimal.java:3003)
> at scala.math.BigDecimal.setScale(BigDecimal.scala:592)
> at org.apache.spark.sql.types.Decimal.set(Decimal.scala:155)
> at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:610)
> at org.apache.spark.sql.types.Decimal$.fromString(Decimal.scala:656)
> {code}
> Decimal.fromString / fromStringANSI fast-fail when the number of integral
> digits (precision - scale of the parsed java.math.BigDecimal) exceeds 38.
> That value is computed as an Int; for '1e2147483647' it is 1 - (-2147483647),
> which overflows to a negative number, so the check is skipped and converting
> the BigDecimal to Decimal throws.
> Other affected inputs: '-1E+2147483647', '12e2147483647', '1.5e2147483647',
> and '1e2147483648' (on JDK 21+, where it parses; throws "Underflow").
> Relatedly, the check also counts integral digits for zero, so a zero with a
> large exponent such as '0e39' or '0e100' is rejected as out of range (NULL,
> or NUMERIC_OUT_OF_SUPPORTED_RANGE under ANSI) although it fits any decimal
> type, while '0e37' returns 0.00. '0e2147483647' returns 0.00 only because of
> the Int overflow above. The code dates back to SPARK-35841 and is the same on
> all maintained branches.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
