[
https://issues.apache.org/jira/browse/ARROW-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16881503#comment-16881503
]
Joris Van den Bossche commented on ARROW-5888:
----------------------------------------------
The Parquet file format has no notion of timezones (only whether the values are
UTC normalized or local time). So to roundtrip timezones, we probably need to
add arrow-specific metadata to the parquet schema (in a similar way as spark
does).
There was some discussion about this in
https://github.com/apache/arrow/pull/4421. [~tpboudreau] do you know if there
was already an issue about this?
> [Python][C++] Parquet write metadata not roundtrip safe for timezone
> timestamps
> -------------------------------------------------------------------------------
>
> Key: ARROW-5888
> URL: https://issues.apache.org/jira/browse/ARROW-5888
> Project: Apache Arrow
> Issue Type: Bug
> Reporter: Florian Jetter
> Priority: Minor
>
> The timezone is not roundtrip safe for timezones other than UTC when storing
> to parquet. Expected behavior would be that the timezone is properly
> reconstructed
> {code:python}
> schema = pa.schema(
> [
> pa.field("no_tz", pa.timestamp('us')),
> pa.field("no_tz", pa.timestamp('us', tz="UTC")),
> pa.field("no_tz", pa.timestamp('us', tz="Europe/Berlin")),
> ]
> )
> buf = pa.BufferOutputStream()
> pq.write_metadata(
> schema,
> buf,
> coerce_timestamps="us"
> )
> pq_bytes = buf.getvalue().to_pybytes()
> reader = pa.BufferReader(pq_bytes)
> parquet_file = pq.ParquetFile(reader)
> parquet_file.schema.to_arrow_schema()
> # Output:
> # no_tz: timestamp[us]
> # utc: timestamp[us, tz=UTC]
> # europe: timestamp[us, tz=UTC]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
