ninglinLiu opened a new pull request, #7215: URL: https://github.com/apache/datafusion-comet/pull/7215
## Which issue does this PR close? Closes #7209. ## Rationale for this change The native string-to-timestamp cast returned NULL (or `CAST_INVALID_INPUT` under ANSI) for several zone suffixes that Spark accepts through `ZoneId.of(id, ZoneId.SHORT_IDS)`: `SHORT_IDS` aliases such as `PST`, region ids without a `/` such as `CET` and `Zulu`, a region id with no space before it (`12:34:56America/Los_Angeles`), and offsets with a seconds field (`+08:00:00`). `Etc/GMT+8` was also caught by the `GMT` prefix rule and read as UTC+8 instead of UTC-8. ## What changes are included in this PR? In `native/spark-expr/src/conversion_funcs/string.rs`: - A new step in `extract_offset_suffix` looks for a zone id that starts with a letter right after a digit or whitespace and resolves it the way `ZoneId.of(id, SHORT_IDS)` does: the `SHORT_IDS` alias first (the full Java table, including the existing `EST`/`MST`/`HST` fixed offsets), then a tz database id through `Tz::from_str`. It runs before the `UTC`/`GMT`/`UT` prefix rule, so `Etc/GMT+8` resolves to the region id. Prefix forms such as `GMT+8` and `UTC-08:00` don't resolve as ids and still go through the prefix rule. - This replaces the separate `EST`/`MST`/`HST` step and the "space plus slash-containing name" step, which the new step covers. - `parse_sign_offset` accepts `+HH:MM:SS` (two digits each, as `java.time.ZoneOffset` does). `Tz` has minute precision, so an offset with a non-zero seconds part still isn't resolved and leaves the value unparsed, as before. Zone ids stay case sensitive, matching `ZoneId.of`. ## How are these changes tested? - New Rust unit test `timestamp_parser_zone_id_suffix_test` covers each form from the issue (`PST`, `CST`, `JST`, `EST`, `CET`, `Zulu`, `America/Los_Angeles` with and without a fraction and no space, `Etc/GMT+8`, `Etc/GMT-8`, `+08:00:00`, `-08:00:00`), checks that `GMT+8` and `UTC-08:00` keep their prefix meaning, and that `pst`, `XYZ` and `+08:00:0` are still rejected. - `cargo test -p datafusion-comet-spark-expr --lib` passes (1297 tests), and `cargo clippy -p datafusion-comet-spark-expr --all-targets -- -D warnings` and `cargo fmt` are clean. - Added queries for these suffixes to `sql-tests/expressions/cast/cast.sql`. I did not run the Spark SQL file suite locally. This PR was prepared with AI assistance and reviewed and tested by the author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
