[
https://issues.apache.org/jira/browse/ARROW-8273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan mercier updated ARROW-8273:
------------------------------------
Description:
Dear,
I have an arrow table and one of columns as an arrow list(size=2)
When I try to convert the table to pandas I got an ArrowNotImplementedError
error
below minimal case:
{code:java}
from pyarrow import Schema, Table, int64, list_, schema, array
fields = [('foo', list_(int64(), 2),)]
sc = schema(fields)
foo_column = [[1,2], [3,4]]
columns = [foo_column]
a_table = Table.from_arrays(arrays=columns, schema=sc)
df = a_table.to_pandas()
---------------------------------------------------------------------------
ArrowNotImplementedError Traceback (most recent call last)
<ipython-input-13-87e390bb8b12> in <module>
----> 1 df2 = a_table.to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/array.pxi in
pyarrow.lib._PandasConvertible.to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
pyarrow.lib.Table._to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py in
table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)
764 _check_data_column_metadata_consistency(all_columns)
765 columns = _deserialize_column_index(table, all_columns, column_indexes)
--> 766 blocks = _table_to_blocks(options, table, categories,
ext_columns_dtypes)
767
768 axes = [columns, index]
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py in
_table_to_blocks(options, block_table, categories, extension_columns)
1099 columns = block_table.column_names
1100 result = pa.lib.table_to_blocks(options, block_table, categories,
-> 1101 list(extension_columns.keys()))
1102 return [_reconstruct_block(item, columns, extension_columns)
1103 for item in result]
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
pyarrow.lib.table_to_blocks()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/error.pxi in
pyarrow.lib.check_status()
ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of
type fixed_size_list<item: int64>[2] is known
{code}
If I try to convert python list (column) to array that segfault
{code:java}
from pyarrow import Schema, Table, int64, list_, schema, array
fields = [('foo', list_(int64(), 2),)]
sc = schema(fields)
foo_column = [[1,2], [3,4]]
columns = [foo_column]
a_table = Table.from_arrays(arrays=columns, schema=sc)
df = a_table.to_pandas(){code}
If I remove the fixed size that works:
{code:java}
from pyarrow import Schema, Table, int64, list_, schema, array
fields = [('foo', list_(int64()),)]
sc = schema(fields)
foo_column = [[1,2], [3,4]]
columns = [array(foo_column)]
a_table = Table.from_arrays(arrays=columns, schema=sc)
df = a_table.to_pandas(){code}
Thanks for your help
was:
Dear,
I have an arrow table and one of columns as an arrow list(size=2)
When I try to convert the table to pandas I got an ArrowNotImplementedError
error
below minimal case:
{code:java}
from pyarrow import Schema, Table, int64, list_, schema, array
fields = [('foo', list_(int64(), 2),)]
sc = schema(fields)
foo_column = [[1,2], [3,4]]
columns = [foo_column]
a_table = Table.from_arrays(arrays=columns, schema=sc)
df = a_table.to_pandas()
---------------------------------------------------------------------------
ArrowNotImplementedError Traceback (most recent call last)
<ipython-input-13-87e390bb8b12> in <module>
----> 1 df2 = a_table.to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/array.pxi in
pyarrow.lib._PandasConvertible.to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
pyarrow.lib.Table._to_pandas()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py in
table_to_blockmanager(options, table, categories, ignore_metadata, types_mapper)
764 _check_data_column_metadata_consistency(all_columns)
765 columns = _deserialize_column_index(table, all_columns, column_indexes)
--> 766 blocks = _table_to_blocks(options, table, categories,
ext_columns_dtypes)
767
768 axes = [columns, index]
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py in
_table_to_blocks(options, block_table, categories, extension_columns)
1099 columns = block_table.column_names
1100 result = pa.lib.table_to_blocks(options, block_table, categories,
-> 1101 list(extension_columns.keys()))
1102 return [_reconstruct_block(item, columns, extension_columns)
1103 for item in result]
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
pyarrow.lib.table_to_blocks()
~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/error.pxi in
pyarrow.lib.check_status()
ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of
type fixed_size_list<item: int64>[2] is known
{code}
Maybe I need to convert python list (column) to array
Thanks for your help
> Fail to convert an integer list (arrow) to pandas
> -------------------------------------------------
>
> Key: ARROW-8273
> URL: https://issues.apache.org/jira/browse/ARROW-8273
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Jonathan mercier
> Priority: Minor
>
> Dear,
>
> I have an arrow table and one of columns as an arrow list(size=2)
> When I try to convert the table to pandas I got an ArrowNotImplementedError
> error
> below minimal case:
> {code:java}
> from pyarrow import Schema, Table, int64, list_, schema, array
> fields = [('foo', list_(int64(), 2),)]
> sc = schema(fields)
> foo_column = [[1,2], [3,4]]
> columns = [foo_column]
> a_table = Table.from_arrays(arrays=columns, schema=sc)
> df = a_table.to_pandas()
> ---------------------------------------------------------------------------
> ArrowNotImplementedError Traceback (most recent call last)
> <ipython-input-13-87e390bb8b12> in <module>
> ----> 1 df2 = a_table.to_pandas()
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/array.pxi in
> pyarrow.lib._PandasConvertible.to_pandas()
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
> pyarrow.lib.Table._to_pandas()
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py
> in table_to_blockmanager(options, table, categories, ignore_metadata,
> types_mapper)
> 764 _check_data_column_metadata_consistency(all_columns)
> 765 columns = _deserialize_column_index(table, all_columns, column_indexes)
> --> 766 blocks = _table_to_blocks(options, table, categories,
> ext_columns_dtypes)
> 767
> 768 axes = [columns, index]
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/pandas_compat.py
> in _table_to_blocks(options, block_table, categories, extension_columns)
> 1099 columns = block_table.column_names
> 1100 result = pa.lib.table_to_blocks(options, block_table, categories,
> -> 1101 list(extension_columns.keys()))
> 1102 return [_reconstruct_block(item, columns, extension_columns)
> 1103 for item in result]
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/table.pxi in
> pyarrow.lib.table_to_blocks()
> ~/somewhere/foo/venv/lib64/python3.7/site-packages/pyarrow/error.pxi in
> pyarrow.lib.check_status()
> ArrowNotImplementedError: No known equivalent Pandas block for Arrow data of
> type fixed_size_list<item: int64>[2] is known
> {code}
>
> If I try to convert python list (column) to array that segfault
> {code:java}
> from pyarrow import Schema, Table, int64, list_, schema, array
> fields = [('foo', list_(int64(), 2),)]
> sc = schema(fields)
> foo_column = [[1,2], [3,4]]
> columns = [foo_column]
> a_table = Table.from_arrays(arrays=columns, schema=sc)
> df = a_table.to_pandas(){code}
>
> If I remove the fixed size that works:
> {code:java}
> from pyarrow import Schema, Table, int64, list_, schema, array
> fields = [('foo', list_(int64()),)]
> sc = schema(fields)
> foo_column = [[1,2], [3,4]]
> columns = [array(foo_column)]
> a_table = Table.from_arrays(arrays=columns, schema=sc)
> df = a_table.to_pandas(){code}
>
> Thanks for your help
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
