viirya opened a new pull request, #6841: URL: https://github.com/apache/datafusion-comet/pull/6841
## Which issue does this PR close? Closes #6840. ## Rationale for this change Spark parses `cast(string as decimal)` with `new java.math.BigDecimal(str.trim)`, which accepts every char for which `Character.isDigit(char)` is true, in both the mantissa and the exponent. Comet only normalized fullwidth digits, so strings like '٣' (U+0663) or '1e३' returned NULL, or raised `CAST_INVALID_INPUT` under ANSI, where Spark returns a value. ## What changes are included in this PR? - Replace `normalize_fullwidth_digits` with `normalize_unicode_digits`, which maps every non-ASCII Unicode `Nd` digit in the BMP to ASCII. The digits sit in 36 contiguous runs of ten, so it uses a table of each run's zero code point plus a binary search. This set is identical under JDK 17 (Unicode 13) and JDK 21 (Unicode 15). - Supplementary-plane digits (e.g. U+1D7CE) stay rejected: `BigDecimal` scans UTF-16 code units and a surrogate is not a digit. - The pure-ASCII fast path is unchanged, so there is no extra allocation for ASCII input. - Error behavior is unchanged: NULL in LEGACY/TRY, `CAST_INVALID_INPUT` / `NUMERIC_VALUE_OUT_OF_RANGE` in ANSI, with the original string in `CAST_INVALID_INPUT` messages. - Integral, floating-point and date casts are left alone, since Spark only accepts ASCII digits there. - Update the cast compatibility guide. ## How are these changes tested? - Rust unit tests covering every digit run and its boundaries, plus `parse_string_to_decimal` values and error kinds checked against JDK 17 `BigDecimal`. - New SQL file tests `cast/cast_string_to_decimal_unicode_digits.sql` (non-ANSI, including int/bigint/double/date casts of the same strings staying NULL) and `cast/cast_string_to_decimal_unicode_digits_ansi.sql` (values, `CAST_INVALID_INPUT` for a supplementary digit, `NUMERIC_VALUE_OUT_OF_RANGE` on overflow). Both fail on main and pass with this change. This pull request and its description were written by Isaac. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
