Yevgeni Litvin created ARROW-6849:
-------------------------------------
Summary: Can not read a list of items type
Key: ARROW-6849
URL: https://issues.apache.org/jira/browse/ARROW-6849
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.15.0
Reporter: Yevgeni Litvin
Attachments: test_bad_parquet.tgz
A field having a type of list-of-ints can not be read using
{{parrow.parquet.read_table}} function.
This happens only in pyarrow 0.15.0. When downgrading to 0.14.1, the issue is
not observed.
pyspark version: 2.4.4[^test_bad_parquet.tgz]
Minimal snippet to reproduce the issue:
{code:java}
import pyarrow.parquet as pq
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, IntegerType, ArrayType,
Row
output_url = '/tmp/test_bad_parquet'
spark = SparkSession.builder.getOrCreate()
schema = StructType([StructField('int_fixed_size_list',
ArrayType(IntegerType(), False), False)])
rows = [Row(int_fixed_size_list=[1, 2, 3])]
dataframe = spark.createDataFrame(rows,
schema).write.mode('overwrite').parquet(output_url)
pq.read_table(output_url)
{code}
I get an error:
{code:java}
Traceback (most recent call last):
File "/home/yevgeni/uatc/dataset-toolkit/repro_failure.py", line 13, in
<module>
pq.read_table(output_url)
File
"/home/yevgeni/uatc/.petastorm3.6/lib/python3.6/site-packages/pyarrow/parquet.py",
line 1281, in read_table
use_pandas_metadata=use_pandas_metadata)
File
"/home/yevgeni/uatc/.petastorm3.6/lib/python3.6/site-packages/pyarrow/parquet.py",
line 1137, in read
use_pandas_metadata=use_pandas_metadata)
File
"/home/yevgeni/uatc/.petastorm3.6/lib/python3.6/site-packages/pyarrow/parquet.py",
line 605, in read
table = reader.read(**options)
File
"/home/yevgeni/uatc/.petastorm3.6/lib/python3.6/site-packages/pyarrow/parquet.py",
line 253, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 1136, in
pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 78, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column data for field 0 with type list<item: int32
not null> is inconsistent with schema list<element: int32 not null>Process
finished with exit code 1
{code}
Column data for field 0 with type list<item: int32 not null> is inconsistent
with schema list<element: int32 not null>
A parquet store, as generated by the snippet is attached.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
