andygrove opened a new issue, #354:
URL: https://github.com/apache/datafusion-spark/issues/354
### Describe the bug
`to_utc_timestamp` uses the wrong offset for some times on the day that
daylight saving time starts, and it rejects time zone IDs such as `GMT+1` that
Spark accepts.
`adjust_to_utc_time` in `datetime/to_utc_timestamp.rs` treats the local time
as if it were a UTC instant and looks up the zone's offset at that instant
(`offset_from_utc_datetime`). On the day the clocks change, that instant can
fall on the other side of the change from the local time. It also parses the
zone with chrono's `Tz` rather than `TimeZoneRules`, which `from_utc_timestamp`
uses.
### To Reproduce
```sql
SELECT to_utc_timestamp('2024-03-10 05:00:00', 'America/Los_Angeles');
SELECT to_utc_timestamp('2024-01-15 12:00:00', 'GMT+1');
```
| Query | Spark 4.2.0 | datafusion-spark |
|---|---|---|
| `to_utc_timestamp('2024-03-10 05:00:00', 'America/Los_Angeles')` |
`2024-03-10 12:00:00` | `2024-03-10T13:00:00` |
| `to_utc_timestamp('2024-01-15 12:00:00', 'GMT+1')` | `2024-01-15 11:00:00`
| `Execution error: to_utc_timestamp: invalid timezone 'GMT+1'` |
`05:00` on 2024-03-10 is PDT (UTC-7), but `05:00` UTC is before the change
at `10:00` UTC, so the PST offset (UTC-8) is used. Local times from `03:00` to
`09:59` that day are off by an hour. For comparison,
`from_utc_timestamp('2024-01-15 12:00:00', 'GMT+1')` already returns
`2024-01-15T13:00:00`.
### Expected behavior
The same results as Spark, which resolves the local time with
`LocalDateTime.atZone`. `TimeZoneRules::local_to_utc` in `datetime/timezone.rs`
already does that, including Spark's handling of gaps and overlaps, and is the
inverse of the `adjust_to_local_time` that `from_utc_timestamp` uses. Parsing
the zone with `TimeZoneRules::parse` would also accept the zone IDs that
`from_utc_timestamp` accepts.
### Additional context
Found while reviewing for #353, which tracks moving functions onto the
shared cast and time zone code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]