andygrove opened a new issue, #4682: URL: https://github.com/apache/datafusion-comet/issues/4682
### Describe the bug `CometUnixTimestamp` (`spark/src/main/scala/org/apache/comet/serde/datetime.scala`) gives inconsistent signals for `TimestampNTZType` input: - `getUnsupportedReasons` states `TimestampNTZType is not supported because Comet incorrectly applies timezone conversion to TimestampNTZ values`. - `isSupportedInputType` returns `true` for `TimestampNTZType`, and `getSupportLevel` reports `Compatible()` for it. So the EXPLAIN-time reason and the runtime behavior disagree. If the native path really does apply session-timezone conversion to `TIMESTAMP_NTZ` (as the documented divergence on `unix_timestamp` in `docs/source/contributor-guide/spark_expressions_support.md` suggests), then Comet produces a different value than Spark for non-UTC session timezones. If it does not, the reason text is stale. This is the `unix_timestamp` / `UnixTimestamp` analog of the hour/minute/second TimestampNTZ tz-conversion bug tracked in #3180. ### Steps to reproduce With a non-UTC `spark.sql.session.timeZone`, evaluate `unix_timestamp` (and `to_unix_timestamp`) on a `TIMESTAMP_NTZ` column and compare Comet against Spark. ### Expected behavior The EXPLAIN-time reason must match runtime behavior. If the native path applies session-timezone conversion to `TIMESTAMP_NTZ`, the support level for that input type should be `Incompatible(Some(...))` (cross-referencing #3180); if it does not, the stale reason text should be removed and the inconsistency resolved. ### Additional context Split out from #4502 (item 1), surfaced by the `audit-comet-expression` skill run in #4448. Cross-references #3180 (same tz-conversion divergence for hour/minute/second on TimestampNTZ). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
