adriangb opened a new issue, #11039:
URL: https://github.com/apache/arrow-rs/issues/11039
**Describe the bug**
`string_to_datetime` in `arrow-cast/src/parse.rs`, and therefore every
`Utf8`/`LargeUtf8`/`Utf8View` → `Timestamp(_, Some(tz))` cast, fails for
wall-clock readings that fall on a daylight saving transition of a **named**
timezone:
- the hour repeated by a "fall back" transition (ambiguous), and
- the hour skipped by a "spring forward" transition (nonexistent).
The failure is `Error parsing timestamp from '...': error computing timezone
offset`. Under `CastOptions { safe: true }` the affected values silently become
`NULL`. It affects both a string without a zone that is interpreted in the
target timezone, and a string that carries the zone explicitly
(`'2024-03-10T02:30:00 America/New_York'`).
This is the string-parsing sibling of #11037. The cause is the same: three
call sites in `string_to_datetime` resolve the local time with
`LocalResult::single()`, which is `None` for both `LocalResult::Ambiguous` and
`LocalResult::None`:
```rust
return timezone
.from_local_datetime(&datetime)
.single()
.ok_or_else(|| err("error computing timezone offset"));
// ... (twice more, including for the parsed `Tz` suffix)
```
**To Reproduce**
arrow-cast 59.3.0, arrow-array with the `chrono-tz` feature:
```rust
use std::sync::Arc;
use arrow_array::{Array, ArrayRef, StringArray};
use arrow_array::timezone::Tz;
use arrow_cast::parse::string_to_datetime;
use arrow_cast::{cast_with_options, CastOptions};
use arrow_schema::{DataType, TimeUnit};
fn main() {
let tz: Tz = "America/New_York".parse().unwrap();
for s in [
"2024-11-01T00:00:00", // unambiguous
"2024-11-03T01:30:00", // ambiguous (fall back)
"2024-03-10T02:30:00", // nonexistent (spring
forward)
"2024-03-10T02:30:00 America/New_York", // nonexistent, zone given
in the string
] {
println!("{s:<40} -> {:?}", string_to_datetime(&tz, s).map(|d|
d.to_rfc3339()));
}
let strings: ArrayRef = Arc::new(StringArray::from(vec![
"2024-11-01T00:00:00", "2024-11-03T01:30:00", "2024-03-10T02:30:00",
]));
let to = DataType::Timestamp(TimeUnit::Second,
Some("America/New_York".into()));
let strict = CastOptions { safe: false, ..Default::default() };
println!("Utf8 -> Timestamp(tz) safe=false: {:?}",
cast_with_options(&strings, &to, &strict).map(|a| a.len()));
let out = cast_with_options(&strings, &to, &CastOptions { safe: true,
..Default::default() }).unwrap();
let shown: Vec<String> = (0..out.len())
.map(|i| arrow_cast::display::array_value_to_string(&out,
i).unwrap())
.collect();
println!("Utf8 -> Timestamp(tz) safe=true: {shown:?}");
}
```
```
2024-11-01T00:00:00 -> Ok("2024-11-01T00:00:00-04:00")
2024-11-03T01:30:00 -> Err(ParseError("Error parsing
timestamp from '2024-11-03T01:30:00': error computing timezone offset"))
2024-03-10T02:30:00 -> Err(ParseError("Error parsing
timestamp from '2024-03-10T02:30:00': error computing timezone offset"))
2024-03-10T02:30:00 America/New_York -> Err(ParseError("Error parsing
timestamp from '2024-03-10T02:30:00 America/New_York': error computing timezone
offset"))
Utf8 -> Timestamp(tz) safe=false: Err(ParseError("Error parsing timestamp
from '2024-11-03T01:30:00': error computing timezone offset"))
Utf8 -> Timestamp(tz) safe=true: ["2024-11-01T00:00:00-04:00", "", ""]
```
**Expected behavior**
The same resolution as #11037, which records the PostgreSQL and DuckDB
outputs and links to the source of both:
- ambiguous → the **later** instant (post-transition offset):
`2024-11-03T01:30:00` → `2024-11-03T01:30:00-05:00`;
- nonexistent → shift **forward** by the gap (pre-transition offset):
`2024-03-10T02:30:00` → `2024-03-10T03:30:00-04:00`.
**Additional context**
- Once https://github.com/apache/arrow-rs/pull/11038 lands, arrow-cast is
inconsistent with itself: casting the naive **timestamp** `2024-03-10T02:30:00`
to `Timestamp(_, Some("America/New_York"))` returns `03:30:00-04:00`, while
casting the **string** `'2024-03-10T02:30:00'` to the same type still errors.
Same wall-clock value, same target type, different result depending on the
source type.
- In DataFusion this is the difference between
`'2024-03-10T02:30:00'::timestamp::timestamptz` (works after 11038) and
`'2024-03-10T02:30:00'::timestamptz` (still fails) under a named session
timezone. DataFusion's `timestamps.slt` currently asserts that `TIMESTAMPTZ
'2023-03-12 02:00:00 America/Los_Angeles'` is an error, so that expectation
flips when this is fixed.
- Kept separate from #11037 because `string_to_datetime` is public API and
generic over `T: TimeZone`, so the fix has a different shape:
`from_local_datetime(..).latest()` for the overlap, and for the gap a probe of
`from_local_datetime(dt - 24h).earliest()` to recover the pre-transition offset
followed by `from_utc_datetime`. Three call sites plus tests for the
zone-suffix form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
