[
https://issues.apache.org/jira/browse/ARROW-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Angus Hollands updated ARROW-13151:
-----------------------------------
Description:
Given the following table
{code:java}
data = {"root": [[{"addr": {"this": 3, "that": 3}}]]}
table = pa.Table.from_pydict(data)
{code}
reading the nested column leads to an {{pyarrow.lib.ArrowInvalid}} error:
{code:java}
pq.write_table(table, "/tmp/table.parquet")
file = pq.ParquetFile("/tmp/table.parquet")
array = file.read(["root.list.item.addr.that"])
{code}
Traceback:
{code:java}
Traceback (most recent call last):
File "....", line 21, in <module>
array = file.read(["root.list.item.addr.that"])
File
"/home/angus/.mambaforge/envs/awkward/lib/python3.9/site-packages/pyarrow/parquet.py",
line 383, in read
return self.reader.read_all(column_indices=column_indices,
File "pyarrow/_parquet.pyx", line 1097, in
pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: List child array invalid: Invalid: Struct child array
#0 does not match type field: struct<that: int64> vs struct<that: int64, this:
int64>
{code}
It's possible that I don't quite understand this properly - am I doing
something wrong?
was:
Given the following table
{code:java}
data = {"root": [[{"addr": {"this": 3, "that": 3}}]]}
table = pa.Table.from_pydict(data)
{code}
reading the nested column leads to an `pyarrow.lib.ArrowInvalid` error:
{code}
pq.write_table(table, "/tmp/table.parquet")
file = pq.ParquetFile("/tmp/table.parquet")
array = file.read(["root.list.item.addr.that"])
{code}
Traceback:
{code}
Traceback (most recent call last):
File "....", line 21, in <module>
array = file.read(["root.list.item.addr.that"])
File
"/home/angus/.mambaforge/envs/awkward/lib/python3.9/site-packages/pyarrow/parquet.py",
line 383, in read
return self.reader.read_all(column_indices=column_indices,
File "pyarrow/_parquet.pyx", line 1097, in
pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: List child array invalid: Invalid: Struct child array
#0 does not match type field: struct<that: int64> vs struct<that: int64, this:
int64>
{code}
It's possible that I don't quite understand this properly - am I doing
something wrong?
> [Python] Unable to read single child field of struct column from Parquet
> ------------------------------------------------------------------------
>
> Key: ARROW-13151
> URL: https://issues.apache.org/jira/browse/ARROW-13151
> Project: Apache Arrow
> Issue Type: Bug
> Components: Parquet, Python
> Reporter: Angus Hollands
> Priority: Major
>
> Given the following table
> {code:java}
> data = {"root": [[{"addr": {"this": 3, "that": 3}}]]}
> table = pa.Table.from_pydict(data)
> {code}
> reading the nested column leads to an {{pyarrow.lib.ArrowInvalid}} error:
> {code:java}
> pq.write_table(table, "/tmp/table.parquet")
> file = pq.ParquetFile("/tmp/table.parquet")
> array = file.read(["root.list.item.addr.that"])
> {code}
> Traceback:
> {code:java}
> Traceback (most recent call last):
> File "....", line 21, in <module>
> array = file.read(["root.list.item.addr.that"])
> File
> "/home/angus/.mambaforge/envs/awkward/lib/python3.9/site-packages/pyarrow/parquet.py",
> line 383, in read
> return self.reader.read_all(column_indices=column_indices,
> File "pyarrow/_parquet.pyx", line 1097, in
> pyarrow._parquet.ParquetReader.read_all
> File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
> pyarrow.lib.ArrowInvalid: List child array invalid: Invalid: Struct child
> array #0 does not match type field: struct<that: int64> vs struct<that:
> int64, this: int64>
> {code}
> It's possible that I don't quite understand this properly - am I doing
> something wrong?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
