andygrove opened a new issue, #6849: URL: https://github.com/apache/datafusion-comet/issues/6849
## What / Why Inside a native plan, every `TimestampType` value has to carry the Arrow label `"UTC"`, and every `TimestampNTZType` value no label at all (see "The invariant" in `docs/source/contributor-guide/timezones.md`). A native expression that returns another label passes any test that only projects its result, because the scan and shuffle boundaries cast each column back to its declared type. The mistake only shows up when the result is compared with another timestamp or fed to another native expression. #6328 (`timestamp_seconds`) and #6330 (`date_trunc`) were both this bug. The timezone audit in #6335 found them, and each fix added a test for its own expression, but nothing checks the others. ## What to add A SQL file test, or one per expression, that for each native expression returning a timestamp: - compares the result with a timestamp column (`=` and `<`), and feeds it to `hour` and to a cast to `STRING` - runs in `UTC`, `Etc/UTC` and a zone with DST such as `America/Los_Angeles`, through `ConfigMatrix` - uses `expect_native(<name>)`, because a dispatched expression returns Spark's answer from inside a Comet operator The expressions that run natively today and return `TIMESTAMP` or `TIMESTAMP_NTZ` are `timestamp_seconds`, `date_trunc`, casts to `TIMESTAMP` and `TIMESTAMP_NTZ`, and, with `allowIncompatible`, `from_utc_timestamp`, `to_utc_timestamp` and `convert_timezone`. Expressions that run through the codegen dispatcher get their label from the dispatcher's output schema, so one dispatched case is enough to cover those. Follows up on #6335. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
