bphillips-exos opened a new issue, #35259:
URL: https://github.com/apache/arrow/issues/35259

   ### Describe the enhancement requested
   
   There doesn't seem to be a way to convert a pandas column using the new 
pyarrow backend to a DictionaryArray in arrow. Both categorical and 
non-categorical columns of ArrowDtypes fail conversion to Table with different 
errors. 
   
   Versions:
   * pandas 2.0.0
   * pyarrow 11.0.0
   
   Platform: OSX
   
   ```python
   import pyarrow as pa
   import pandas as pd
   
   schema = pa.schema([
       pa.field(
           name='category',
           type=pa.dictionary(index_type=pa.int8(), value_type=pa.string())
       ),
   ])
   
   df = pd.DataFrame({'category': ['a']})
   
   # works
   pa.Table.from_pandas(df, schema=schema)
   
   # works
   df2 = df.assign(category=df['category'].astype('category'))
   pa.Table.from_pandas(df2, schema=schema)
   
   # fails
   df3 = df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())))
   pa.Table.from_pandas(df3, schema=schema)
   
   # fails
   df4 = 
df.assign(category=df['category'].astype(pd.ArrowDtype(pa.string())).astype('category'))
   pa.Table.from_pandas(df4, schema=schema)
   ```
   
   df3 fails with 
   ```
       pa.Table.from_pandas(df3, schema=schema)
     File "pyarrow/table.pxi", line 3572, in pyarrow.lib.Table.from_pandas
     File "pyarrow/table.pxi", line 3657, in pyarrow.lib.Table.from_arrays
     File "pyarrow/table.pxi", line 1421, in pyarrow.lib._sanitize_arrays
     File "pyarrow/array.pxi", line 347, in pyarrow.lib.asarray
     File "pyarrow/table.pxi", line 523, in pyarrow.lib.ChunkedArray.cast
     File 
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/compute.py",
 line 391, in cast
       return call_function("cast", [arr], options)
     File "pyarrow/_compute.pyx", line 560, in pyarrow._compute.call_function
     File "pyarrow/_compute.pyx", line 355, in pyarrow._compute.Function.call
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 121, in pyarrow.lib.check_status
   pyarrow.lib.ArrowNotImplementedError: Unsupported cast from string to 
dictionary using function cast_dictionary
   ```
   
   df4 fails with
   ```
       pa.Table.from_pandas(df4, schema=schema)
     File "pyarrow/table.pxi", line 3557, in pyarrow.lib.Table.from_pandas
     File 
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
 line 611, in dataframe_to_arrays
       arrays = [convert_column(c, f)
     File 
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
 line 611, in <listcomp>
       arrays = [convert_column(c, f)
     File 
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
 line 598, in convert_column
       raise e
     File 
"/Users/bphillips/workspace/exos-chronos/.venv/lib/python3.10/site-packages/pyarrow/pandas_compat.py",
 line 592, in convert_column
       result = pa.array(col, type=type_, from_pandas=True, safe=safe)
     File "pyarrow/array.pxi", line 310, in pyarrow.lib.array
     File "pyarrow/array.pxi", line 2608, in 
pyarrow.lib.DictionaryArray.from_arrays
     File "pyarrow/array.pxi", line 320, in pyarrow.lib.array
     File "pyarrow/array.pxi", line 39, in pyarrow.lib._sequence_to_array
     File "pyarrow/error.pxi", line 144, in 
pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow/error.pxi", line 100, in pyarrow.lib.check_status
   pyarrow.lib.ArrowInvalid: ("Could not convert <pyarrow.StringScalar: 'a'> 
with type pyarrow.lib.StringScalar: did not recognize Python value type when 
inferring an Arrow data type", 'Conversion failed for column category with type 
category')
   ```
   
   ### Component(s)
   
   Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to