crusaderky opened a new issue, #33727:
URL: https://github.com/apache/arrow/issues/33727
### Describe the bug, including details regarding any error messages,
version, and platform.
pandas 1.5.2
pyarrow 10.0.1
If you convert a pandas Series with dtype `string[pyarrow]` to `category`,
the categories will be `string[pyarrow]`. So far, so good.
However, when you try writing the resulting object to parquet, PyArrow fails
as it does not recognize its own datatype.
## Reproducer
```python
>>> import pandas as pd
>>> df = pd.DataFrame({"x": ["foo", "bar", "foo"], dtype="string[pyarrow]")
>>> df.dtypes.x
string[pyarrow]
>>> df = df.astype("category")
>>> df.dtypes.x
CategoricalDtype(categories=['bar', 'foo'], ordered=False)
>>> df.dtypes.x.categories.dtype
string[pyarrow]
>>> df.to_parquet("foo.parquet")
pyarrow.lib.ArrowInvalid: ("Could not convert <pyarrow.StringScalar: 'bar'>
with type pyarrow.lib.StringScalar: did not recognize Python value type when
inferring an Arrow data type", 'Conversion failed for column x with type
category')
```
## Workaround
```python
df = df.astype(
{
k: pd.CategoricalDtype(v.categories.astype(object))
for k, v in df.dtypes.items()
if isinstance(v, pd.CategoricalDtype)
and v.categories.dtype == "string[pyarrow]"
}
)
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
