adriangb opened a new issue, #11037:
URL: https://github.com/apache/arrow-rs/issues/11037
**Describe the bug**
Casting a `Timestamp(_, None)` array to a `Timestamp(_, Some(tz))` with a
**named** timezone fails for every value whose wall-clock reading falls on a
daylight saving transition:
- the hour repeated by a "fall back" transition (ambiguous), and
- the hour skipped by a "spring forward" transition (nonexistent).
With `CastOptions { safe: false }` the whole cast fails with `Cast error:
Cannot cast timezone to different timezone`; with `safe: true` the affected
values silently become `NULL`. Unambiguous readings and fixed-offset timezones
(`+08:00`) are fine.
The cause is `adjust_timestamp_to_timezone` in `arrow-cast/src/cast/mod.rs`:
```rust
let adjust = |o| {
let local = as_datetime::<T>(o)?;
let offset = to_tz.offset_from_local_datetime(&local).single()?;
T::from_naive_datetime(local - offset.fix(), None)
};
```
`LocalResult::single()` is `None` for both `LocalResult::Ambiguous` and
`LocalResult::None`, so both cases collapse into the failure path.
**To Reproduce**
arrow-cast 59.3.0, arrow-array with the `chrono-tz` feature:
```rust
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, TimestampSecondArray};
use arrow_cast::{cast_with_options, CastOptions};
use arrow_schema::{DataType, TimeUnit};
fn main() {
// Naive wall-clock readings, to be interpreted in America/New_York:
// 2024-11-01T00:00:00 unambiguous
// 2024-11-03T01:30:00 ambiguous (fall back: 01:30 happens twice)
// 2024-03-10T02:30:00 nonexistent (spring forward: 02:30 never
happens)
let naive: ArrayRef = Arc::new(TimestampSecondArray::from(vec![
1_730_419_200, 1_730_597_400, 1_710_037_800,
]));
let to = DataType::Timestamp(TimeUnit::Second,
Some("America/New_York".into()));
let strict = CastOptions { safe: false, ..Default::default() };
println!("safe=false: {:?}", cast_with_options(&naive, &to,
&strict).map(|a| a.len()));
let lenient = CastOptions { safe: true, ..Default::default() };
let out = cast_with_options(&naive, &to, &lenient).unwrap();
let shown: Vec<String> = (0..out.len())
.map(|i| arrow_cast::display::array_value_to_string(&out,
i).unwrap())
.collect();
println!("safe=true: {shown:?}");
}
```
```
safe=false: Err(CastError("Cannot cast timezone to different timezone"))
safe=true: ["2024-11-01T00:00:00-04:00", "", ""]
```
**Expected behavior**
Both values resolve to an instant instead of failing. PostgreSQL 17.11 and
DuckDB 1.5.2 agree exactly on how:
```sql
SET TimeZone = 'America/New_York';
SELECT '2024-11-03T01:30:00'::timestamp::timestamptz AS ambiguous,
'2024-03-10T02:30:00'::timestamp::timestamptz AS nonexistent;
```
```
ambiguous | nonexistent
------------------------+------------------------
2024-11-03 01:30:00-05 | 2024-03-10 03:30:00-04
```
- **Ambiguous**: pick the **later** instant, i.e. the post-transition
(standard) offset. In chrono terms `LocalResult::Ambiguous(_, later)` → `later`.
- **Nonexistent**: shift **forward** by the size of the gap, which is the
same as interpreting the reading with the pre-transition offset.
So `[1_730_419_200, 1_730_597_400, 1_710_037_800]` should become the
instants `[1_730_433_600, 1_730_615_400, 1_710_055_800]`, displayed as
`2024-11-01T00:00:00-04:00`, `2024-11-03T01:30:00-05:00`,
`2024-03-10T03:30:00-04:00`.
**Additional context**
- Reported against DataFusion as
https://github.com/apache/datafusion/issues/25084. I first prototyped a
DataFusion-side workaround (https://github.com/apache/datafusion/pull/25115,
now closed) but the kernel is the right place: every arrow-rs consumer hits
this, and DataFusion has cast paths that call the kernel directly.
- I will open a PR that changes `adjust_timestamp_to_timezone` to the
resolution above. Adding an explicit policy to `CastOptions` (like Arrow C++'s
`AssumeTimezoneOptions` with `ambiguous`/`nonexistent` = raise/earliest/latest)
would be a breaking change to a public struct, so I am proposing the
deterministic behaviour as the fix now; a policy option can follow in a major
release if anyone needs `earliest` or `raise`.
- `string_to_datetime` in `arrow-cast/src/parse.rs` has the same `.single()`
pattern in three places (`Error parsing timestamp ...: error computing timezone
offset`). That is a separate, generic-`TimeZone` code path and is not part of
this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
