andygrove opened a new issue, #5012:
URL: https://github.com/apache/datafusion-comet/issues/5012
### Describe the bug
With `spark.sql.ansi.enabled=true`, `CAST(string AS date)` must raise
`CAST_INVALID_INPUT` for a malformed input. Comet returns `NULL` instead when
the string has the shape of a date but is not a real calendar date, e.g.
`'2016-13-01'`, `'2016-02-30'`, `'2016-01-32'`.
Strings that fail to parse structurally (`'abc'`) do throw correctly, and
`CAST(string AS timestamp)` throws correctly for the same inputs, so this is
specific to the date target.
Root cause looks like the fast path in `date_parser`
(`native/spark-expr/src/conversion_funcs/string.rs:1948-1962`): for a canonical
10-byte `yyyy-mm-dd` shape it returns `Ok(ymd_to_epoch_day(..).map(..))`
directly, so an invalid calendar date yields `Ok(None)` → null, bypassing
`return_result`, which is the only place the `EvalMode::Ansi` error is raised.
The comment above that block asserts that "Comet already returns null for that
in every eval mode ... matching what the general parser does here", which is
what the repro contradicts.
This is the Tier 4 item from #4180 (does `spark.sql.ansi.enabled` propagate
beyond the paths where it was originally wired up).
### Steps to reproduce
The expression has to run over a real column, otherwise Spark constant-folds
it and Comet never executes.
```scala
val path = "/tmp/ansi_date"
spark.sql("SELECT s FROM VALUES ('2016-13-01'),('2016-02-30'),('2016-01-01')
AS v(s)")
.write.mode("overwrite").parquet(path)
spark.sql(s"CREATE OR REPLACE TEMPORARY VIEW t AS SELECT * FROM
parquet.`$path`")
spark.conf.set("spark.sql.ansi.enabled", "true")
spark.conf.set("spark.comet.enabled", "false")
spark.sql("SELECT cast(s as date) FROM t").collect()
// org.apache.spark.SparkDateTimeException: [CAST_INVALID_INPUT] The value
'2016-13-01'
// of the type "STRING" cannot be cast to "DATE" because it is malformed.
spark.conf.set("spark.comet.enabled", "true")
spark.sql("SELECT cast(s as date) FROM t").show()
// +-------------+
// |CAST(s AS DATE)|
// +-------------+
// | NULL|
// | NULL|
// | 2016-01-01|
// +-------------+
```
`to_date(s)` goes through the same cast and behaves the same way.
The consequence is not limited to a missing exception. A filter on the cast
silently returns rows where Spark fails the query:
```sql
SELECT count(*) FROM t WHERE cast(s as date) > date'2000-01-01'
-- spark: SparkDateTimeException
-- comet: 1
```
Behaviour that is already correct and should stay that way:
* `spark.sql.ansi.enabled=false` — both return `NULL` (verified, matches).
* `try_cast(s as date)` under ANSI — both return `NULL` (verified, matches).
* `cast(s as timestamp)` / `cast(s as timestamp_ntz)` under ANSI — both
throw (verified).
### Expected behavior
Under `EvalMode::Ansi`, `cast(string as date)` should raise
`CAST_INVALID_INPUT` for an invalid calendar date, matching the behaviour for
structurally malformed strings and for the timestamp target.
### Additional context
* Reproduced on `apache/main` @ `0761e549a`, default Maven profile (Spark
4.1.2, where ANSI is on by default), macOS aarch64, debug build. Applies to
Spark 3.x too whenever ANSI is enabled explicitly.
* An adjacent, much smaller discrepancy found in the same sweep:
`cast(string as boolean)` under ANSI throws the right error class and message
but as `SparkNumberFormatException` where Spark throws `SparkRuntimeException`.
Not filed separately; mentioning here in case it is worth folding into the same
fix.
* Found while auditing the config inventory in #4180.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
