andygrove opened a new issue, #5749: URL: https://github.com/apache/datafusion-comet/issues/5749
### Describe the bug Found while reviewing #5682. This is not caused by that PR, it reproduces at its merge base `4219fc793`. `timestamp_ntz_parser_inner` maps a `None` from `parse_to_timestamp_info` to `Ok(None)` regardless of eval mode ([`native/spark-expr/src/conversion_funcs/string.rs:1871`](https://github.com/apache/datafusion-comet/blob/main/native/spark-expr/src/conversion_funcs/string.rs#L1871)), so an input the decoder rejects returns NULL even under ANSI. The `TIMESTAMP` path does not have this problem: it funnels through `timestamp_parser_with_tz`, whose `timestamp.is_none()` branch raises `InvalidInputInCastToDatetime`. The reachable trigger today is the year-range guard at `string.rs:1147`: ```rust if !(-290309..=294248).contains(&year) { return Ok(None); } ``` A 6-digit year satisfies `RE_DAY` (and Spark's `isValidDigits`, which allows up to 6 year digits for a timestamp) but can still fall outside that range, so the guard fires after shape recognition has already succeeded. ### To reproduce Use a Parquet-backed column so the cast is not constant-folded: ```sql SET spark.sql.session.timeZone=UTC; SET spark.sql.ansi.enabled=true; CREATE TABLE y USING parquet AS SELECT '294249-01-01' AS s; SELECT CAST(s AS TIMESTAMP_NTZ) FROM y; ``` Spark 4.1.3 raises: ``` [CAST_INVALID_INPUT] The value '294249-01-01' of the type "STRING" cannot be cast to "TIMESTAMP_NTZ" because it is malformed. Correct the value as per the syntax, or change its target type. Use `try_cast` to tolerate malformed input and return NULL instead. SQLSTATE: 22018 ``` Comet returns NULL. `SELECT CAST(s AS TIMESTAMP) FROM y` on the same input raises correctly, so the two timestamp targets disagree with each other. Verified by adding `294249-01-01`, `294249-01-01 00:00:00` and `-290310-01-01` to `sparkSegmentRuleMalformedTimestamps` in `CometNativeCastSuite` and running both segment-rule tests on Spark 4.1.3. The `TIMESTAMP` test passes and the `TIMESTAMP_NTZ` test fails with `Comet should have failed with [CAST_INVALID_INPUT] ...`. ### Expected behavior `CAST(string AS TIMESTAMP_NTZ)` should raise `CAST_INVALID_INPUT` under ANSI for any input it rejects, matching both Spark and Comet's own `TIMESTAMP` path. ### Additional context The `None` arm at `string.rs:1871` should raise under ANSI the same way the `local_datetime_to_micros` arm directly above it already does. Worth adding ANSI column coverage for out-of-range 6-digit years, positive and negative, against both timestamp targets while fixing it. This also matters for the two fraction boundary guards added in #5682 (`string.rs:1157` and `:1904`). They are unreachable today because every accepted pattern is ASCII-only, but they return `Ok(None)` on the same path, so the "a future regex change cannot restore the UTF-8 boundary panic" invariant they exist to protect would come back as a silently swallowed ANSI error on the NTZ side rather than a panic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
