[
https://issues.apache.org/jira/browse/ARROW-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609932#comment-17609932
]
Joris Van den Bossche commented on ARROW-17636:
-----------------------------------------------
Thanks, that's very useful!
So the issue here is that when using a nullable integer data type in pandas,
which is a pandas extension type, pyarrow tries to recreate that data type on
the "to_pandas()" conversion based on the information that we store in the
table's metadata.
Now, because the write/read operation with partitioning, the data that is being
read again now has a dictionary type, while we do preserve the metadata. As a
result, we get metadata and actual data that are no longer fully matching (the
metadata says the type was an integer extension dtype, the actual data is now
dictionary type), and this currently fails in the conversion.
Reproducing this in another way (explicitly changing this type in the table to
get such a table with out-of-sync metadata and actual types):
{code:python}
df = pd.DataFrame({"col": pd.array([1, 2, 3, 1, 2], dtype="Int64")})
table = pa.table(df)
table2 = table.set_column(0, "col", table["col"].dictionary_encode())
>>> table2.schema
col: dictionary<values=int64, indices=int32, ordered=0>
-- schema metadata --
pandas: '{"index_columns": [{"kind": "range", "name": null, "start": 0, "' + 390
>>> table2.to_pandas()
....
~/scipy/repos/arrow/python/pyarrow/pandas_compat.py in _reconstruct_block(item,
columns, extension_columns)
778 raise ValueError("This column does not support to be
converted "
779 "to a pandas ExtensionArray")
--> 780 pd_ext_arr = pandas_dtype.__from_arrow__(arr)
781 block = _int.make_block(pd_ext_arr, placement=placement)
782 else:
~/miniconda3/envs/arrow-dev/lib/python3.8/site-packages/pandas/core/arrays/numeric.py
in __from_arrow__(self, array)
53 # test_from_arrow_type_error raise for string, but allow
54 # through itemsize conversion GH#31896
---> 55 rt_dtype = pandas_dtype(array.type.to_pandas_dtype())
56 if rt_dtype.kind not in ["i", "u", "f"]:
57 # Could allow "c" or potentially disallow float<->int
conversion,
~/scipy/repos/arrow/python/pyarrow/types.pxi in
pyarrow.lib.DataType.to_pandas_dtype()
NotImplementedError: dictionary<values=int64, indices=int32, ordered=0>
{code}
The error is indirectly being raised from the pandas code, but I have to check
a bit whether we should fix this on the pyarrow or pandas side (we might need
to catch this error on the pyarrow side, and in that case not try to convert to
the pandas extension dtype but use the normal conversion path).
> [Python] Converting Table to pandas raises NotImplementedError (when table
> previously saved as partitioned parquet dataset)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-17636
> URL: https://issues.apache.org/jira/browse/ARROW-17636
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 9.0.0
> Environment: Docker container, based on continuumio/anaconda3
> Python 3.9.12
> PyArrow 9.0.0
> Reporter: Roberto Lobo
> Priority: Major
> Attachments: bug.py
>
>
> When converting a table in which one of the column's type is of
> DictionaryType (values=int32, indices=int32, ordered=0) the conversion to
> pandas DataFrame fails with:
> NotImplementedError: dictionary<values=int32, indices=int32, ordered=0>
> The dictionary has this conversion not implmented yet.
> This DictionaryType is used as type when using one of the columns (Int64) as
> one of the parquet's dataset partition columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
