Florian Jetter created ARROW-10146:
--------------------------------------
Summary: [Python] Parquet metadata to_dict raises attribute error
Key: ARROW-10146
URL: https://issues.apache.org/jira/browse/ARROW-10146
Project: Apache Arrow
Issue Type: Bug
Components: Python
Reporter: Florian Jetter
h2. Description
When accessing rowgroup metadata and trying to convert it to a dict using the
method {{to_dict}} I sometimes receive an Attribute error.
This can be consistently produced with an empty dataframe (see example below)
but I have also seen it already for non empty dataframes. I couldn't track down
what makes the non-empty Attribute errors special, therefore the example below.
h2. Expected behaviour
I would expect the to_dict to always, consistently return a dictionary with the
appropriate metadata and statistics irregardless of the file content.
h2. Minimal Example
{code:python}
import pandas as pd
import pyarrow as pa
import pyarrow.parquet as pq
df = pd.DataFrame({"col": [1]}).head(0)
table = pa.Table.from_pandas(df)
buf = pa.BufferOutputStream()
pq.write_table(table, buf)
reader = pa.BufferReader(buf.getvalue())
parquet_file = pq.ParquetFile(reader)
# Raises Attribute Error
parquet_file.metadata.to_dict()
{code}
h3. Traceback
{code:java}
~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx
in pyarrow._parquet.FileMetaData.to_dict()
~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx
in pyarrow._parquet.RowGroupMetaData.to_dict()
~/miniconda3/envs/kartothek-dev/lib/python3.7/site-packages/pyarrow/_parquet.pyx
in pyarrow._parquet.ColumnChunkMetaData.to_dict()
AttributeError: 'NoneType' object has no attribute 'to_dict'
{code}
h3. Versions
{code:java}
In [28]: pa.__version__
Out[28]: '1.0.1'
In [29]: pd.__version__
Out[29]: '1.0.5'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
