snmvaughan opened a new pull request, #7522: URL: https://github.com/apache/datafusion-comet/pull/7522
## Which issue does this PR close? Closes #7187. ## Rationale for this change [SPARK-60119](https://issues.apache.org/jira/browse/SPARK-60119) (apache/spark#59335) is in Spark 4.0.5, 4.1.4 and 4.2.1. `Decimal.changePrecision` no longer calls `BigDecimal.setScale` when the value's magnitude already decides the result: a value below 0.01 ulp of the target scale rounds as 0.01 ulp of the same sign. With `spark.sql.legacy.allowNegativeScaleOfDecimal=true`, that changes the negative-scale case of #6966. `'1e-2147483647'` cast to `DECIMAL(10,-2)` returns 0 in every eval mode, where earlier releases throw `ArithmeticException: Underflow`. The native cast still returns NULL, or raises `CAST_INVALID_INPUT` under ANSI, because `parse_string_to_decimal` rejects a scale difference that does not fit in an int. The fix leaves nothing in the plan for Comet to detect, so Comet picks the behavior by Spark version, as #7171 does for SPARK-58821 and SPARK-60118. ## What changes are included in this PR? - `SparkFixes.castsTinyStringToDecimalZero` holds the version check: 4.0.5+, 4.1.4+, 4.2.1+ and 4.3+, the same releases as the other two fixes. - `cast`: a new `Cast` proto field, `decimal_tiny_to_zero`, which `CometCast` sets from that check. When it is set, `parse_string_to_decimal` returns 0 for a scale difference below the int range, for either sign. A difference above the int range is a value too large for any precision and stays an error. Without the option the cast fails as before, since earlier releases fail there. - `CometNativeCastSuite`: `(ansi) cast string to negative-scale decimal fails when rescaling overflows an int` and `(ansi) failing cast of a literal raises Spark's own exception when reached` expect Spark to fail on `'1e-2147483647'`, so they now run only on releases without SPARK-60119. The second one has nothing to compare on the newer releases, where Spark folds that literal to `0.00`. - The `cast` audit entry and the string-to-decimal note in the user guide describe the new behavior. ## How are these changes tested? - Rust unit tests: - `test_parse_string_to_decimal_tiny_value_to_negative_scale` covers both signs and a 38-digit mantissa with the option off and on. It also checks that the option leaves alone a difference above the int range and one that fits in an int. - `test_cast_tiny_string_to_negative_scale_decimal` casts an array to `Decimal128` and `Decimal256` in all three eval modes, with the option taken from `SparkCastOptions`. - `SparkFixesSuite` pins the version boundaries of the new check together with the other two. - `CometNativeCastSuite`: - `cast StringType to DecimalType sets the decimal options for the running Spark version` checks both proto fields against `SparkFixes`. - `cast string far below a negative target scale to decimal returns 0 (SPARK-60119)` compares Comet with Spark in ANSI and non-ANSI mode on a Spark with the fix: three strings far below the ulp, one that rounds to one ulp, and NULL. None of the Spark versions CI builds against (3.4.3, 3.5.9, 4.0.4, 4.1.3, 4.2.0) has the fix, so it is skipped there until one does, and the two tests it complements keep running. - `cargo fmt --check`, spotless and Prettier are clean. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
