bharadwaj-pendyala opened a new pull request, #7217: URL: https://github.com/apache/datafusion-comet/pull/7217
## Which issue does this PR close? Closes #7003. ## Rationale for this change Ersa-tech filed #7003 after spotting it while working on #6969. I ran their numbers on main first, through a Parquet table and `cast(cast(d AS timestamp) AS long)` with ANSI off. Spark gives `9223372036854` for `1e16` and `9.3e12`, and `-9223372036855` for `-1e16`. Comet gives NULL for all three. Spark's non-ANSI cast is `(d * MICROS_PER_SECOND).toLong` in `doubleToTimestamp` (and `(long)` in codegen), so it saturates and only NaN and infinity become NULL. Float widens to double first and follows the same rule. `try_cast` is different: Spark runs the ANSI path and catches the error, so the overflow stays NULL there. ## What changes are included in this PR? - `cast_float_to_timestamp_array` gets its own Legacy branch. NaN and infinity still map to NULL, and any other value is `(val * MICROS_PER_SECOND as f64) as i64`. Rust's `as` saturates the same way the JVM's `(long)` does. - Try keeps calling `float_timestamp_micros`, so it still returns NULL on overflow. ANSI doesn't change. ## How are these changes tested? - New Rust test `test_cast_float_to_timestamp_legacy_saturates` covers `1e16`, `-1e16`, `9.3e12`, `f64::MAX` and NaN for Legacy and Try, plus `f32::MAX` and `f32::MIN`. On main it fails at the first value, where Legacy returns NULL. - New SQL file `expressions/cast/cast_float_to_timestamp.sql` reads the same values from Parquet `double` and `float` columns and checks both `cast` and `try_cast` against Spark. On main it fails at rows 3 to 6 (Spark `9223372036854` / `-9223372036855`, Comet NULL). With this change it passes. I ran it locally with the default profile, Spark 4.1.3. - `cargo test -p datafusion-comet-spark-expr --lib` (1297 passed), `cargo clippy --all-targets -p datafusion-comet-spark-expr -- -D warnings` and `cargo fmt --check` are clean. The existing `cast DoubleType to TimestampType` suite test already feeds `Double.MaxValue` through this cast and passes on main, which is why I added the SQL file instead of leaning on it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
