bphillips-exos opened a new issue, #35259:
URL: https://github.com/apache/arrow/issues/35259
### Describe the enhancement requested
There doesn't seem to be a way to convert a pandas column using the new
pyarrow backend to a DictionaryArray in arrow. Both categorical and
non-categorical columns of ArrowDtypes fail conversion to Table with different
errors.
Versions:
* pandas 2.0.0
* pyarrow 11.0.0
Platform: OSX
```python
import pyarrow as pa
import pandas as pd
schema = pa.schema([
pa.field(
name='category',
type=pa.dictionary(index_type=pa.int8(), value_type=pa.string())
),
])
df = pd.DataFrame({'category': ['a']})
# works
pa.Table.from_pandas(df, schema=schema)
# works
df2 = df.assign(category=df['category'].astype('category'))
pa.Table.from_pandas(df2, schema=schema)
# fails
df3 = df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())))
pa.Table.from_pandas(df3, schema=schema)
# fails
df4 =
df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())).astype('category'))
pa.Table.from_pandas(df4, schema=schema)
```
df3 fails with
```
pa.Table.from_pandas(df3, schema=schema)
File "pyarrow/table.pxi", line 3572, in pyarrow.lib.Table.from_pandas
File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
File "pyarrow/table.pxi", line 1421, in pyarrow.lib._sanitize_arrays
File "pyarrow/array.pxi", line 347, in pyarrow.lib.asarray
File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
File
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/compute.py",
line 391, in cast
return call_function("cast", [arr], options)
File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
pyarrow.lib.ArrowNotImplementedError: Unsupported cast from string to
dictionary using function cast_dictionary
```
df4 fails with
```
pa.Table.from_pandas(df4, schema=schema)
File "pyarrow/table.pxi", line 3557, in pyarrow.lib.Table.from_pandas
File
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
line 611, in dataframe_to_arrays
arrays = [convert_column(c, f)
File
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
line 611, in <listcomp>
arrays = [convert_column(c, f)
File
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
line 598, in convert_column
raise e
File
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
line 592, in convert_column
result = pa.array(col, type=type_, from_pandas=True, safe=safe)
File "pyarrow/array.pxi", line 310, in pyarrow.lib.array
File "pyarrow/array.pxi", line 2608, in
pyarrow.lib.DictionaryArray.from_arrays
File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
File "pyarrow/error.pxi", line 144, in
pyarrow.lib.pyarrow_internal_check_status
File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: ("Could not convert <pyarrow.StringScalar: 'a'>
with type pyarrow.lib.StringScalar: did not recognize Python value type when
inferring an Arrow data type", 'Conversion failed for column category with type
category')
```
### Component(s)
Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]