[
https://issues.apache.org/jira/browse/ARROW-13756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17406681#comment-17406681
]
Joris Van den Bossche commented on ARROW-13756:
-----------------------------------------------
Thanks for the report. This is indeed only happening when using a DatetimeIndew
with timezone as the column labels (for a row index it works, as you mentioned
ARROW-1958 fixed this for row index).
A smaller reproducer (it's not needed to have the roundtrip to parquet to
reproduce the error):
{code:python}
df = pd.DataFrame(
np.random.randn(5, 3),
columns=pd.date_range("2021-01-01", "2021-01-3", freq="D", tz="CET")
)
table = pa.table(df)
>>> table.to_pandas()
...
TypeError: data type 'datetimetz' not understood
{code}
In the code to reconstruct the rows Index ({{_extract_index_level}}), we
specifically deal with timezone-aware data. We will need to do something
similar in the code to reconstruct the column Index
({{_reconstruct_columns_from_metadata}}).
> [Python] Error in pandas conversion for datetimetz column index
> ---------------------------------------------------------------
>
> Key: ARROW-13756
> URL: https://issues.apache.org/jira/browse/ARROW-13756
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 5.0.0
> Environment: Ubuntu 21.04
> Reporter: Andreas Wolf
> Priority: Major
>
> The following code fails with:
> {code:java}
> File "[...]/lib/python3.8/site-packages/pyarrow/pandas_compat.py", line 1052,
> in _pandas_type_to_numpy_type
> return np.dtype(pandas_type)
> TypeError: data type 'datetimetz' not understood{code}
> Sample:
> {code:java}
> def run():
> filename = "test.parquet"
> df = pd.DataFrame(
> data=range(31),
> columns=list("A"),
> index=pd.date_range("2021-01-01", "2021-01-31", freq="D", tz="CET"),
> ).T
> table = pa.Table.from_pandas(df)
> pq.write_to_dataset(table, root_path=filename)
> result = pq.read_table(filename).to_pandas()
> return result
> if __name__ == "__main__":
> run()
> {code}
> The code tries to store a dataframe where the columns are timezone aware
> datetimes.
> _Observations_:
> If I remove the *.T* at the end of the dataframe, so that the datatime index
> are rows it is working (but not what I want).
> If I remove the timezone information *tz="CET"* the code is working.
> I assume this bug is related to [Error in pandas conversion for datetimetz
> row index|https://issues.apache.org/jira/browse/ARROW-1958]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
