andygrove opened a new issue, #5069:
URL: https://github.com/apache/datafusion-comet/issues/5069
### What is the problem the feature request solves?
The float/double-to-decimal cast is currently marked `Incompatible` in
`CometCast`
(`spark/src/main/scala/org/apache/comet/expressions/CometCast.scala:409-411,
420-422`), so it only runs with
`spark.comet.expression.Cast.allowIncompatible=true`. One of the divergences
that keeps it incompatible: under ANSI mode, Comet throws for NaN and
+/-Infinity inputs where Spark returns NULL.
Spark 4.1 (`Cast.scala`, both interpreted and codegen paths) wraps
`Decimal(fractional.toDouble(b))` in a catch of `NumberFormatException`, so
`BigDecimal.valueOf(NaN)` results in NULL even under ANSI. Only finite values
that overflow the target precision throw.
Comet (`native/spark-expr/src/conversion_funcs/numeric.rs:907-937`) nulls
NaN/Inf via `unary_opt`, but then, under ANSI, rescans for newly introduced
nulls and raises `NumericValueOutOfRange { value: "NaN" | "inf" }`. The native
unit test `test_cast_float_to_decimal_ansi_nan_errors` (`numeric.rs:1424-1450`)
locks in this incorrect behavior.
### Describe the potential solution
Exempt NaN and +/-Infinity from the ANSI overflow rescan in
`cast_floating_point_to_decimal128` (only finite overflowing values should
error), and invert the native test. This removes one of the blockers to
promoting the cast pair out of `Incompatible`.
### Additional context
Found by an ANSI-mode audit of all native expressions against Spark 4.1.1.
Filed as an enhancement rather than a bug because the cast pair is already
gated behind `allowIncompatible`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
