andygrove opened a new pull request, #4638: URL: https://github.com/apache/datafusion-comet/pull/4638
## Which issue does this PR close? Part of #4596 (the timezone group: `from_utc_timestamp`, `to_utc_timestamp`, `convert_timezone`). ## Rationale for this change All three report `Incompatible` for the same reason: Comet's native timezone parser does not accept Spark's legacy zone forms (`PST`, `GMT+1`, `UTC+1`, ...), so those throw a native parse error (#2013). With `allowIncompatible` unset the whole projection falls back to Spark. They have a real Spark `doGenCode` and supported input/output types, so they are eligible for the `CodegenDispatchFallback` path: route the `Incompatible` result through the JVM codegen dispatcher (Spark's own `doGenCode` inside the Comet pipeline) so the projection stays native, matches Spark exactly, and handles every timezone form. ## What changes are included in this PR? * `CometFromUTCTimestamp`, `CometToUTCTimestamp`, `CometConvertTimezone` mix in `CodegenDispatchFallback`. The native incompatible path stays opt-in via `allowIncompatible`. * `docs/source/user-guide/latest/expressions.md` notes updated: these route through the dispatcher by default and now handle all timezone forms. ## How are these changes tested? New `from_utc_timestamp_dispatch.sql`, `to_utc_timestamp_dispatch.sql`, and `convert_timezone_dispatch.sql`, each with `allowIncompatible` unset and a `ConfigMatrix: spark.sql.session.timeZone=UTC,America/Los_Angeles`. They assert native execution matching Spark across both session zones (confirming the resolved `timeZoneId` survives closure serialization, the concern called out in the issue), and include the legacy `PST` / `GMT+1` forms the native path rejects but the dispatcher handles. The existing `allowIncompatible=true` native-path tests still pass. All run with `CometSqlFileTestSuite` on Spark 3.5. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
