[
https://issues.apache.org/jira/browse/SPARK-60133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
L. C. Hsieh reassigned SPARK-60133:
-----------------------------------
Assignee: L. C. Hsieh
> Decimal.set fails or is slow when the value scale is far from the target scale
> ------------------------------------------------------------------------------
>
> Key: SPARK-60133
> URL: https://issues.apache.org/jira/browse/SPARK-60133
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.5.8, 4.2.0, 4.3.0, 4.1.3, 4.0.4, 4.4.0
> Reporter: L. C. Hsieh
> Assignee: L. C. Hsieh
> Priority: Major
>
> Follow-up of SPARK-60119, which bounds the rescaling in
> `Decimal.changePrecision`.
> `Decimal.set(decimal: BigDecimal, precision: Int, scale: Int)` still calls
> `decimal.setScale(scale, ROUND_HALF_UP)` without bounding the scale change.
> It is used by the JSON, CSV and XML parsers for decimal fields
> (`JacksonParser`, `UnivocityParser`, `StaxXmlParser`), and also by Avro, JDBC
> and `ToNumberParser`. With a `DECIMAL(10,2)` schema:
> - `from_json('{"a": 1e-2147483647}', 'a DECIMAL(10,2)')` and
> `from_csv('1e-2147483647', 'a DECIMAL(10,2)')` fail with
> `ArithmeticException: BigInteger would overflow supported range`, so the
> record is treated as malformed (null in PERMISSIVE, error in FAILFAST), while
> `1e-10000000` correctly gives 0.00.
> - `1e-10000000` takes ~1.4s per value and `1e10000000` ~5s per value;
> exponents around 1e8 take ~40s per value.
> Overflow in this method is reported as
> `NUMERIC_VALUE_OUT_OF_RANGE.WITHOUT_SUGGESTION` with a `roundedValue`, so the
> large side needs a decision on what to report without computing the rounded
> value.
> `Decimal.set(decimal: BigDecimal)` has a similar issue: with
> `spark.sql.legacy.allowNegativeScaleOfDecimal=false` it calls
> `decimal.setScale(0)` for a negative scale, which fails for e.g. a
> `java.math.BigDecimal("1e2147483647")` coming from a non-string input. That
> method is tracked in SPARK-60132; this ticket covers `Decimal.set(decimal:
> BigDecimal, precision: Int, scale: Int)`. The string cast path is guarded by
> the fast-fail in `Decimal.fromString` (see SPARK-60118).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
