L. C. Hsieh created SPARK-60200:
-----------------------------------
Summary: Casting a decimal with a huge negative scale to string or
showing it builds an unbounded plain string
Key: SPARK-60200
URL: https://issues.apache.org/jira/browse/SPARK-60200
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, `DecimalType` has no
lower bound on a negative scale. Casting such a decimal to string under ANSI
(`useDecimalPlainString`, `ToStringBase`) and `Dataset.show` (`ToPrettyString`)
call `Decimal.toPlainString`, which builds the full plain string. For
`1E+2147483647` it fails with `OutOfMemoryError: too large to fit in a String`:
{code:scala}
lit("1e2147483647").cast(DecimalType(1, -2147483647)).cast("string")
spark.range(1).select(lit(new java.math.BigDecimal("1e2147483647"))).show()
{code}
The inner cast succeeds because the precision and scale already match, so
`Decimal.changePrecision` takes its fast path and the value reaches the string
conversion.
SPARK-60119 bounds this only for the `NUMERIC_VALUE_OUT_OF_RANGE` error message
(scientific notation below scale -1000). The threshold could move to a
`Decimal`-level helper shared by these callers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
