tlfua opened a new issue, #7468: URL: https://github.com/apache/datafusion-comet/issues/7468
### Describe the bug Under ANSI mode, Comet evaluates the right side of `=`, `<`, `<=`, `>`, `>=` (and probably `<=>`'s siblings that use `defineCodeGen`) on rows where the left side is NULL. Spark skips the right side on those rows: `EqualTo` and the ordering comparisons use `defineCodeGen`/`nullSafeCodeGen`, so the right operand is only evaluated where the left one is not NULL. When the right side can fail, such as an ANSI cast of a malformed string, Comet raises an error that Spark never raises. This is the same shape as https://github.com/apache/datafusion-comet/issues/7246, which covers arithmetic and `concat` and is being fixed in https://github.com/apache/datafusion-comet/pull/7456. It was raised in review there by @comphead: https://github.com/apache/datafusion-comet/pull/7456#pullrequestreview-5759066188 ### Steps to reproduce With `spark.sql.ansi.enabled=true` and `spark.sql.adaptive.enabled=false` (Spark 4.1.3, Comet built from https://github.com/apache/datafusion-comet/pull/7456): ```sql CREATE TABLE cmp(id int, n int, txt string) USING parquet; INSERT INTO cmp SELECT /*+ REPARTITION(1) */ * FROM VALUES (1, CAST(NULL AS INT), 'bad'), (2, 2, '2') AS t(id, n, txt); SELECT id, n < CAST(txt AS INT) FROM cmp; ``` Comet fails in `CometProject` with: ``` [CAST_INVALID_INPUT] The value 'bad' of the type "STRING" cannot be cast to "INT" because it is malformed. ... ``` ### Expected behavior Same as Spark: `(1, NULL)`, `(2, false)`. The cast is never evaluated for row 1 because `n` is NULL. ### Additional context `SELECT id FROM cmp WHERE n = CAST(txt AS INT)` did not fail in the same run, presumably because the inferred `isnotnull(n)` filter and Comet's lazy `AND` skip the row first. A projection, or any position where no such filter is inferred, does fail. A fix could mark the comparisons the same way https://github.com/apache/datafusion-comet/pull/7456 marks arithmetic, with `NullShortCircuit` over the comparison when the right side can fail. ### Willingness to contribute I can contribute a fix for this bug independently -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
