[
https://issues.apache.org/jira/browse/ARROW-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17609937#comment-17609937
]
Joris Van den Bossche commented on ARROW-17636:
-----------------------------------------------
[~rhlobo] one possible workaround on the short term could be to avoid using
dictionary type for the partitioning fields, which case be done with:
{code}
result_table = pq.read_table(source='xbug.pq',
partitioning=ds.HivePartitioning.discover(infer_dictionary=False))
{code}
Or, alternatively, if you want to use dictionary / category dtype for those
fields, you can remove the metadata before converting to pandas:
{code}
result_table = pq.read_table(source='xbug.pq')
result_table = result_table.replace_schema_metadata()
result_table.to_pandas()
{code}
but this way, also the other columns will not be converted to a pandas nullable
data type, but to plain numpy (non-nullable) integer dtypes.
> [Python] Converting Table to pandas raises NotImplementedError (when table
> previously saved as partitioned parquet dataset)
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: ARROW-17636
> URL: https://issues.apache.org/jira/browse/ARROW-17636
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 9.0.0
> Environment: Docker container, based on continuumio/anaconda3
> Python 3.9.12
> PyArrow 9.0.0
> Reporter: Roberto Lobo
> Assignee: Joris Van den Bossche
> Priority: Major
> Fix For: 10.0.0
>
> Attachments: bug.py
>
>
> When converting a table in which one of the column's type is of
> DictionaryType (values=int32, indices=int32, ordered=0) the conversion to
> pandas DataFrame fails with:
> NotImplementedError: dictionary<values=int32, indices=int32, ordered=0>
> The dictionary has this conversion not implmented yet.
> This DictionaryType is used as type when using one of the columns (Int64) as
> one of the parquet's dataset partition columns.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
