andygrove opened a new pull request, #4753: URL: https://github.com/apache/datafusion-comet/pull/4753
## Which issue does this PR close? Closes #3180. ## Rationale for this change The native `hour`, `minute`, and `second` implementations applied a session timezone offset to every input. That conversion is correct for timezone-aware timestamps (stored in UTC) but wrong for TimestampNTZ values, which already hold local wall-clock time. As a short-term fix these expressions were marked `Incompatible` for TimestampNTZ inputs so they fell back to Spark. This PR implements the follow-up enhancement tracked in the issue: handling TimestampNTZ correctly on the native path so the expressions run in Comet for all supported input types. ## What changes are included in this PR? - Native (`extract_date_part.rs`): detect TimestampNTZ inputs (including dictionary-encoded) and extract the date part directly with no timezone conversion. Timezone-aware timestamps keep the existing UTC to session-timezone shift. - Scala serde: `CometHour`, `CometMinute`, and `CometSecond` no longer special-case TimestampNTZ. They are `Compatible` for all supported input types and no longer need the `Incompatible` reason or the codegen-dispatch fallback. - Retargeted the `native opt-in hint shown for codegen-dispatch path` test in `CometExpressionSuite` to use `from_utc_timestamp`, since `Hour` is no longer `Incompatible` for any input. ## How are these changes tested? - New Rust unit tests in `extract_date_part.rs` covering: timezone-aware conversion, TimestampNTZ direct extraction, timezone independence of the NTZ result, `minute`/`second` on NTZ, the NTZ-detection helper, and dictionary-encoded NTZ input. - `CometTemporalExpressionSuite` `hour/minute/second - timestamp_ntz input` now verifies native execution and Spark-identical results across UTC, America/Los_Angeles, Europe/London, and Asia/Tokyo. Two tests that only covered the previous fallback and `allowIncompatible` behavior were removed as obsolete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
