jorisvandenbossche commented on issue #50667:
URL: https://github.com/apache/arrow/issues/50667#issuecomment-5219654648
> But I don't fully understand when you would end up with a `datetime.UTC`
timezone.
I figured out this part: in my test I was manually converting some pyarrow
data to pandas, but actually when using `pd.read_parquet`, it does some
post-processing after converting the pyarrow table to a pandas dataframe. And
one of those things is to convert pytz timezones, meaning if you did end up
with a pytz UTC timezone, it gets converted to datetime.UTC, but if you already
have zoneinfo UTC timezone, we leave it alone (that's potentially something to
fix in pandas to make more consistent, to always normalize the UTC timezones).
```python
# creating a small parquet file
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
table = pa.table({"col": pd.date_range("2020-01-01", periods=3, tz="UTC")})
pq.write_table(table, "test_utc.parquet")
# reading it with pandas 3.0.5 and pyarrow 25.0
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
zoneinfo.ZoneInfo(key='UTC')
# reading it with pandas 3.0.5 and pyarrow 24.0
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
datetime.timezone.utc
# unless pytz is not installed, then pyarrow already gives a zoneinfo zone
and pandas does not convert it to datetime.UTC
>>> import sys
>>> sys.modules["pytz"] = None
>>> pd.read_parquet("test_utc.parquet")["col"].dtype.tz
zoneinfo.ZoneInfo(key='UTC')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
