[
https://issues.apache.org/jira/browse/ARROW-14104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434368#comment-17434368
]
Sarah Gilmore commented on ARROW-14104:
---------------------------------------
Hi [~jorisvandenbossche],
It actually looks like I was running an older version of pyarrow based on the
output of {{pa.__version__}}. According to pip, I have pyarrow 5.0.0:
{code:java}
// code placeholder
Name: pyarrow
Version: 5.0.0
Summary: Python library for Apache Arrow
Home-page: https://arrow.apache.org/
Author:
Author-email:
License: Apache License, Version 2.0
Location: /usr/local/lib/python3.9/site-packages
Requires: numpy
Required-by: parquet-tools
{code}
But {{pa.__version__}} returns {{'0.17.1'}}. It looks like my system
configuration got messed up, though I'm not sure how. I was able to confirm
that the TimeZone is round-tripped in pyarrow 5.0.0 by creating a virtual
environment with python's venv module and installing pyarrow 5.0.0 there.
I'm sorry for any confusion I caused. I'll close this issue.
Best,
Sarah
> Reading Lists of Timestamps from parquet files in Arrow 5.0.0 fails to
> preserve the TimeZone - unlike in Arrow 4.0.0
> --------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-14104
> URL: https://issues.apache.org/jira/browse/ARROW-14104
> Project: Apache Arrow
> Issue Type: Bug
> Components: C++, Parquet, Python
> Affects Versions: 5.0.0
> Reporter: Sarah Gilmore
> Priority: Minor
> Attachments: exampleArrow4.parq, exampleArrow5.parq
>
>
> In Arrow 4.0.0 it is possible to round-trip the TimeZone property of
> List<Timestamp> columns to and from parquet files:
> {code:java}
> >>> import pyarrow as pa
> >>> import pyarrow.parquet as pq
> >>> import datetime
> >>> column = pa.array([[datetime.datetime(2023, 9, 23, 11)]],
> >>> pa.list_(pa.timestamp('us', 'America/New_York')));
> >>> t = pa.Table.from_arrays([column], names=['TimestampColumn']);
> >>> pq.write_table(t, "example.parq");
> >>> t2 = pq.read_table("example.parq");
> >>> t2
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=America/New_York]>
> child 0, item: timestamp[us, tz=America/New_York]
> {code}
> However, if you read the same parquet file in pyarrow 5.0.0, the TimeZone is
> set to UTC:
> {code:java}
> >>> t3 = pq.read_table("example.parq");
> >>> t3
> pyarrow.Table
> Dates: list<item: timestamp[us, tz=UTC]>
> child 0, item: timestamp[us, tz=UTC]
> {code}
>
> I noticed that the TimeZone is preserved in Arrow 5.0 when reading non-nested
> timestamp columns.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
