viirya opened a new issue, #6840:
URL: https://github.com/apache/datafusion-comet/issues/6840
### Describe the bug
Spark casts a string to decimal with `new java.math.BigDecimal(str.trim)`
(`Decimal.fromString` / `fromStringANSI`). `BigDecimal` accepts any char for
which `Character.isDigit(char)` is true, i.e. every Unicode `Nd` digit in the
Basic Multilingual Plane (Arabic-Indic, Devanagari, Thai, fullwidth, ...), in
both the mantissa and the exponent. Comet's native parser only normalizes
fullwidth digits (U+FF10–U+FF19), so other scripts return NULL. Under ANSI,
Comet raises `CAST_INVALID_INPUT` where Spark returns a value.
### Steps to reproduce
```sql
CREATE TABLE t(s string) USING parquet;
INSERT INTO t VALUES ('٣'), ('१२३.४५'), ('1e٣'); -- U+0663 ARABIC-INDIC
DIGIT THREE, Devanagari digits
SELECT s, try_cast(s AS DECIMAL(10,2)) FROM t;
```
Spark returns `3.00`, `123.45`, `1000.00`. Comet returns NULL for all three.
With `spark.sql.ansi.enabled=true`, `cast(s AS DECIMAL(10,2))` fails in Comet
with `CAST_INVALID_INPUT`.
### Expected behavior
Same results as Spark. For reference, on JDK 17 `new BigDecimal(s)` accepts
BMP `Nd` digits from any script, mixed freely ('1٢e३' is 12000), and rejects
supplementary-plane digits such as U+1D7D0 MATHEMATICAL BOLD DIGIT TWO (a
surrogate pair), as well as `No`/`Nl` characters such as '²' and 'Ⅳ'.
The integral, floating-point and date casts from string are not affected:
Spark only accepts ASCII digits there (`UTF8String.toLong`,
`Double.parseDouble`, `SparkDateTimeUtils.stringToDate`), and Comet already
matches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
