DG47 opened a new pull request, #4065:
URL: https://github.com/apache/iceberg-python/pull/4065
Closes #3833
# Rationale for this change
`ArrowProjectionVisitor.list` rebuilds a list whose element is a struct with
`pa.LargeListArray.from_arrays(list_array.offsets, value_array)`. The offsets
alone cannot express a null list, so every null `list<struct<...>>` came out as
an empty list. The visitor runs on both the read and the write path; on write
the empty list is persisted into the Parquet file, so the null is gone for
every reader, not only for pyiceberg.
Reproduction from the issue (pyarrow 25.0.1):
```
l_struct in=[[{'x': 1}], [], None, [{'x': 3}]] out=[[{'x': 1}], [], [],
[{'x': 3}]]
l_int in=[[1], [], None, [3]] out=[[1], [], None, [3]]
```
This PR rebuilds the list with `pa.Array.from_buffers`, reusing the validity
and offsets buffers (and the array offset) of the original list and swapping in
the projected element array. Passing `mask=list_array.is_null()` to
`from_arrays` would fix the unsliced case, but Arrow rejects a mask together
with sliced offsets (`Null bitmap with offsets slice not supported`) for
`LargeListArray`, so sliced batches would start raising. Reusing the buffers
keeps the array offset and avoids that.
The integration test `test_null_list_and_map` asserted the corrupted value
with the correct assertion commented out; the correct assertion is restored.
## Are these changes tested?
- New unit test `test__to_requested_schema_null_list_of_structs` in
`tests/io/test_pyarrow.py`, parametrized over `pa.list_` and `pa.large_list`,
covering a full batch and a sliced one. It fails on `main` (`At index 2 diff:
[] != None`) and passes with this change.
- Restored the real assertion in
`tests/integration/test_reads.py::test_null_list_and_map`.
- `make test` and `make lint` pass locally.
## Are there any user-facing changes?
Null lists of structs are now preserved on read and write instead of being
turned into empty lists.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
