andygrove commented on issue #4502: URL: https://github.com/apache/datafusion-comet/issues/4502#issuecomment-4695966236
Revisited the high-priority items. Since this issue was filed from the #4448 audit, these datetime expressions migrated to the JVM codegen dispatcher and `spark.comet.exec.scalaUDF.codegen.enabled` flipped to default `true`, which changes the picture: - **Item 1 (`CometUnixTimestamp` NTZ):** Real, but the predicate was right and the reason text was stale. The native `unix_timestamp` path has a dedicated TimestampNTZ branch that divides raw micros with no timezone conversion, matching Spark. Fixed the misleading `getUnsupportedReasons` text in #4645. - **Item 2 (move `CometDateFormat` gating into `getSupportLevel`):** Now counterproductive. With the dispatcher on by default and producing Spark-identical results, marking non-UTC + whitelisted formats `Incompatible` would route them to full Spark fallback under the default config, a regression. Recommend closing this sub-item. - **Item 3 (Group A "Compatible while silently falling back"):** Largely moot. The dispatcher defaults to enabled, so `Compatible()` is accurate for the default config; when disabled, `convert` returns `None` with a `withFallbackReason` visible in EXPLAIN. `CometCodegenDispatch` deliberately omits the flag from `getCompatibleNotes`. The only stale artifact was `CometDateFormat`'s note saying the dispatcher is "disabled (default)", corrected in #4645. - **Item 4 (`CometMakeTimestamp` ANSI):** Resolved by the migration. `CometMakeTimestamp` is now `CometCodegenDispatch[MakeTimestamp]`, so Spark's own `doGenCode` (which honors `failOnError`) runs in the kernel and throws under ANSI; when the dispatcher is off it falls back to Spark. Correct in both configs. Recommend closing this sub-item. Net: items 1 and the date_format doc fix are in #4645; items 2 and 4 can be closed as obsolete, and item 3 is resolved aside from that doc fix. The medium-priority items (5, 6) are untouched. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
