fenfeng9 commented on issue #50470:
URL: https://github.com/apache/arrow/issues/50470#issuecomment-4952158701
A null parent struct masks its child slots. Therefore, an underlying Arrow
child-array slot may be null when the parent struct is null, even if the child
field is non-nullable. This is valid because the child is logically absent. The
Parquet writer incorrectly rejects this case.
#### Reproduce
```python
import pyarrow as pa
import pyarrow.parquet as pq
value_field = pa.field("value", pa.int32(), nullable=False)
# The physical child slot is null.
value_column = pa.array([None], type=pa.int32())
# True means that the corresponding parent struct is null.
parent_is_null = pa.array([True])
array = pa.StructArray.from_arrays(
arrays=[value_column],
fields=[value_field],
mask=parent_is_null,
)
print("logical array:", array.to_pylist())
print("physical child:", array.field("value").to_pylist())
pq.write_table(pa.table({"data": array}), "output.parquet")
```
#### Result
```python
logical array: [None]
physical child: [None]
Traceback (most recent call last):
File "/path/to/test.py", line 22, in <module>
pq.write_table(pa.table({"data": array}), "output.parquet")
~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/site-packages/pyarrow/parquet/core.py", line 2015, in
write_table
writer.write_table(table, row_group_size=row_group_size)
~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/path/to/site-packages/pyarrow/parquet/core.py", line 1180, in
write_table
self.writer.write_table(table, row_group_size=row_group_size)
~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pyarrow/_parquet.pyx", line 2397, in
pyarrow._parquet.ParquetWriter.write_table
File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 'value' is declared non-nullable but
contains nulls
```
#### Analysis
This appears to be a regression introduced by #44921.
The check uses `leaf_array.null_count()` without accounting for null parent
structs, so it incorrectly rejects child nulls masked by a null parent.
https://github.com/apache/arrow/blob/5815dc0b4e3d9239828b47d427c9759fbc506e09/cpp/src/parquet/column_writer.cc#L1432-L1442
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
