L. C. Hsieh created SPARK-60133:
-----------------------------------
Summary: Decimal.set fails or is slow when the value scale is far
from the target scale
Key: SPARK-60133
URL: https://issues.apache.org/jira/browse/SPARK-60133
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: L. C. Hsieh
Follow-up of SPARK-60119, which bounds the rescaling in
`Decimal.changePrecision`.
`Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)` still calls
`decimal.setScale(scale, ROUND_HALF_UP)` without bounding the scale change. It
is used by the JSON, CSV and XML parsers for decimal fields (`JacksonParser`,
`UnivocityParser`, `StaxXmlParser`), and also by Avro, JDBC and
`ToNumberParser`. With a `DECIMAL(10,2)` schema:
- `from_json('{"a": 1e-2147483647}', 'a DECIMAL(10,2)')` and
`from_csv('1e-2147483647', 'a DECIMAL(10,2)')` fail with `ArithmeticException:
BigInteger would overflow supported range`, so the record is treated as
malformed (null in PERMISSIVE, error in FAILFAST), while `1e-10000000`
correctly gives 0.00.
- `1e-10000000` takes ~1.4s per value and `1e10000000` ~5s per value; exponents
around 1e8 take ~40s per value.
Overflow in this method is reported as
`NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION` with a `roundedValue`, so the
large side needs a decision on what to report without computing the rounded
value.
`Decimal.set(decimal: BigDecimal)` has a similar issue: with
`spark.sql.legacy.allowNegativeScaleOfDecimal=false` it calls
`decimal.setScale(0)` for a negative scale, which fails for e.g. a
`java.math.BigDecimal("1e2147483647")` coming from a non-string input. The
string cast path is guarded by the fast-fail in `Decimal.fromString` (see
SPARK-60118).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
