[
https://issues.apache.org/jira/browse/ARROW-5952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16896980#comment-16896980
]
Joris Van den Bossche commented on ARROW-5952:
----------------------------------------------
[~nugend] Thanks for the report!
Looking at your example, I noticed that the original table and the table read
from the file are different in the following way (building on the code from
above):
{code}
new_table = pa.ipc.open_file('bar').read_all()
original_column = table.column(0)
new_column = new_table.column(0)
{code}
{code}
>>> original_column
<pyarrow.lib.ChunkedArray object at 0x7f06cc5d1e40>
[
-- dictionary:
0 nulls
-- indices:
[]
]
>>> new_column
<pyarrow.lib.ChunkedArray object at 0x7f06c7029540>
[
]
>>> len(original_column)
0
>>> len(new_column)
0
>>> len(original_column.chunks)
1
>>> len(new_column.chunks)
0
{code}
So both columns have the same type and a length of 0, but the original column
has 1 chunk of length 0, while the table read from file trough IPC has a column
consisting of a ChunkedArray with 0 chunks.
So in that way, we can reproduce the segfault with a small snippet as follows:
{code}
>>> import pyarrow as pa
>>> array_0_chunks = pa.chunked_array([], pa.dictionary(pa.int8(), pa.null()))
>>> table = pa.table({'a': array_0_chunks})
>>> table.to_pandas()
>>>
>>>
Segmentation fault (core dumped)
{code}
I am not fully sure if the IPC reading is supposed to create an array with 0
chunks, but I suppose that such a ChunkedArray with 0 chunks is a valid object
(for simple types as int64 it also does not segfault).
But the segfault on the pandas conversion is for sure a bug (it seems that
other arrow->python conversions such as {{to_pydict}} seems to work, so it
might be specific to the pandas code)
> [Python] Segfault when reading empty table with category as pandas dataframe
> ----------------------------------------------------------------------------
>
> Key: ARROW-5952
> URL: https://issues.apache.org/jira/browse/ARROW-5952
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Affects Versions: 0.14.0, 0.14.1
> Environment: Linux 3.10.0-327.36.3.el7.x86_64
> Python 3.6.8
> Pandas 0.24.2
> Pyarrow 0.14.0
> Reporter: Daniel Nugent
> Priority: Major
>
> I have two short sample programs which demonstrate the issue:
> {code:java}
> import pyarrow as pa
> import pandas as pd
> empty = pd.DataFrame({'foo':[]},dtype='category')
> table = pa.Table.from_pandas(empty)
> outfile = pa.output_stream('bar')
> writer = pa.RecordBatchFileWriter(outfile,table.schema)
> writer.write(table)
> writer.close()
> {code}
> {code:java}
> import pyarrow as pa
> pa.ipc.open_file('bar').read_pandas()
> Segmentation fault
> {code}
> My apologies if this was already reported elsewhere, I searched but could not
> find an issue which seemed to refer to the same behavior.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
