Kyle Beauchamp created ARROW-9827:
-------------------------------------
Summary: pandas.read_parquet fails for wide parquet files and
pyarrow 1.0.X
Key: ARROW-9827
URL: https://issues.apache.org/jira/browse/ARROW-9827
Project: Apache Arrow
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: Kyle Beauchamp
I recently tried to update my pyarrow from 0.17.1 to 1.0.0 and I'm encountering
a serious bug where wide DataFrames fail during pandas.read_parquet. Small
parquet files (m=10000) read correctly, medium files (m=40000) fail with a "Bus
Error: 10", and large files (m=100000) completely hang. I've tried python
3.8.5, pandas 1.0.5, pyarrow 1.0.0, and OSX 10.14. I recently tried to update
my pyarrow from 0.17.1 to 1.0.0 and I'm encountering a serious bug where wide
DataFrames fail during read_parquet. Small parquet files (m=10000) read
correctly, medium files (m=40000) fail with a "Bus Error: 10", and large files
(m=100000) completely hang. I've tried python 3.8.5, pandas 1.0.5, pyarrow
1.0.0, and OSX 10.14.
The driver code and output is below:
{code:java}
import pandas as pd
import numpy as np
import sys
filename = "test.parquet"
n = 10
m = int(sys.argv[1])
print(m)
x = np.zeros((n, m))
x = pd.DataFrame(x, columns=[f"A_{i}" for i in range(m)])
x.to_parquet(filename)
y = pd.read_parquet(filename, engine='pyarrow'){code}
{code:java}
time python test_pyarrow.py 10000
real 0m4.018s user 0m5.286s sys 0m0.514s
time python test_pyarrow.py 40000
40000
Bus error: 10
{code}
On a pyarrow 0.17.1 environment, the 40,000 case completes in 8 seconds.
This was cross-posted on the pandas tracker as well:
[https://github.com/pandas-dev/pandas/issues/35846]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
