FreakingMind opened a new pull request, #7634:
URL: https://github.com/apache/datafusion-comet/pull/7634

   ## Which issue does this PR close?
   
   Closes #7001.
   
   ## Rationale for this change
   
   With ANSI on, `cast(d AS timestamp)` for a NaN or infinite double or float 
fails in both engines, but with different errors. Spark throws 
`SparkDateTimeException` saying `The value Infinity of the type "DOUBLE" cannot 
be cast to "TIMESTAMP"`. Comet threw `SparkNumberFormatException` saying `The 
value 'inf' of the type "STRING"`. The error class and SQLSTATE already 
matched, only the exception class and the text were off.
   
   Spark does this the same way in 3.5 through 4.2. `doubleToTimestampAnsi` 
calls the `Double` overload of `invalidInputInCastToDatetimeError`. A float is 
widened first, so the message says DOUBLE for float input too.
   
   ## What changes are included in this PR?
   
   - The native cast now returns `InvalidInputInCastToDatetime` for NaN and 
infinities, with the value spelled the way Spark spells it (`NaN`, `Infinity`, 
`-Infinity`).
   - Both error converter shims have a new case for a DOUBLE source that calls 
Spark's `Double` overload. The string case is unchanged.
   - I renamed `float_timestamp_overflow_text` to `double_sql_text`, since the 
overflow error and the new error both use it now.
   
   ## How are these changes tested?
   
   - A new SQL test, `cast_float_to_timestamp_ansi.sql`, compares the exception 
class and the message with Spark for double and float input. It fails on main 
with `SparkNumberFormatException did not equal SparkDateTimeException`.
   - New cases in `SparkErrorConverterSuite` for a DOUBLE and a STRING source.
   - An updated Rust unit test.
   
   I ran the SQL test and the converter suite on Spark 4.1 and 3.5.
   
   The existing `CometNativeCastSuite` float and double tests didn't catch 
this, because their generated data hits `CAST_OVERFLOW` on `MaxValue` before it 
reaches NaN.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to