hanke580 opened a new issue, #11698:
URL: https://github.com/apache/arrow-rs/issues/11698
### Describe the bug
`arrow_csv::Writer` and `arrow_json::Writer` format timestamps with a time
zone through `arrow_cast::display`.
`write_timestamp()` (`arrow-cast/src/display.rs`, around line 770) uses
`date.to_rfc3339_opts(SecondsFormat::AutoSi, true)`. chrono computes the
wall time with the exact offset, but RFC 3339
has no seconds in offsets, so the offset is printed **rounded to whole
minutes**. The two parts of the written value
then disagree.
For `1960-06-01T12:00:00Z` in `Africa/Monrovia` (offset −00:44:30 at the
time), the writers produce
`1960-06-01T11:15:30-00:45`, which is `12:00:30Z`: 30 seconds later than the
stored instant. Reading the file back
(with arrow-csv/arrow-json, DataFusion, or any other tool) gives the shifted
instant. There is no error.
This applies to every zone whose offset was not a whole number of minutes:
Africa/Monrovia until 1972, Europe/Dublin
until 1916, Europe/Paris until 1911, and every zone's local mean time (LMT)
before standard time, e.g.
`America/New_York` before 1883. Parquet and IPC are not affected.
Writing the instant in UTC (`Z`), or the offset with seconds (`-00:44:30`,
which chrono and most parsers accept even
though it is outside RFC 3339), would be lossless.
### To Reproduce
```rust
use std::sync::Arc;
use arrow::array::{ArrayRef, TimestampMicrosecondArray};
use arrow::datatypes::{DataType, Field, Schema, TimeUnit};
use arrow::record_batch::RecordBatch;
fn main() {
// 1960-06-01T12:00:00Z; Africa/Monrovia's offset was then -00:44:30
let ts =
TimestampMicrosecondArray::from(vec![-302_443_200_000_000i64]).with_timezone("Africa/Monrovia");
let schema = Arc::new(Schema::new(vec![Field::new("ts",
DataType::Timestamp(TimeUnit::Microsecond,
Some("Africa/Monrovia".into())), false)]));
let batch = RecordBatch::try_new(schema, vec![Arc::new(ts) as
ArrayRef]).unwrap();
let mut csv = Vec::new();
arrow::csv::Writer::new(&mut csv).write(&batch).unwrap();
println!("csv: {}", String::from_utf8(csv).unwrap());
let mut json = arrow::json::LineDelimitedWriter::new(Vec::new());
json.write(&batch).unwrap();
json.finish().unwrap();
println!("json: {}", String::from_utf8(json.into_inner()).unwrap());
}
```
(`arrow = { version = "60.0.0", features = ["chrono-tz", "csv", "json"] }`)
```
csv: ts
1960-06-01T11:15:30-00:45
json: {"ts":"1960-06-01T11:15:30-00:45"}
```
`11:15:30-00:45` is `12:00:30Z`. The array holds `12:00:00Z`.
The same through DataFusion (`datafusion-cli` 55.1.0, main 290ad15),
including a read back:
```sql
CREATE TABLE t AS SELECT arrow_cast(arrow_cast('1960-06-01T12:00:00Z',
'Timestamp(Microsecond, Some("UTC"))'),
'Timestamp(Microsecond,
Some("Africa/Monrovia"))') AS ts;
COPY t TO 'o/t.csv' STORED AS CSV; -- 1960-06-01T11:15:30-00:45
CREATE EXTERNAL TABLE c (ts TIMESTAMPTZ) STORED AS CSV LOCATION 'o/t.csv'
OPTIONS ('format.has_header' 'true');
SELECT ts FROM c; -- 1960-06-01T12:00:30
```
| zone, stored instant | written | read back (UTC) |
|---|---|---|
| Africa/Monrovia, 1960-06-01T12:00:00Z | `1960-06-01T11:15:30-00:45` |
12:00:30 |
| Europe/Dublin, 1900-06-01T12:00:00Z | `1900-06-01T11:34:39-00:25` |
11:59:39 |
| Europe/Paris, 1900-06-01T12:00:00Z | `1900-06-01T12:09:21+00:09` |
12:00:21 |
| America/New_York, 1880-06-01T12:00:00Z | `1880-06-01T07:03:58-04:56` |
11:59:58 |
| Africa/Monrovia, 2000-06-01T12:00:00Z | `2000-06-01T12:00:00Z` | 12:00:00 |
### Expected behavior
The written text denotes the same instant as the array value, e.g.
`1960-06-01T12:00:00Z` or `1960-06-01T11:15:30-00:44:30`.
### Additional context
- arrow 60.0.0. `write_timestamp` in `arrow-cast/src/display.rs` is
unchanged on `main` (last change to the file
2026-09-29).
- In DataFusion this also affects `INSERT INTO` a CSV/JSON-backed table
under a session time zone: with
`SET datafusion.execution.time_zone = 'Africa/Monrovia'`, `INSERT … VALUES
(1, TIMESTAMPTZ '1960-06-01T12:00:00Z')`
stores `1960-06-01T11:15:30-00:45` in the file and reads back `12:00:30Z`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
