rahulsmahadev opened a new pull request, #4436:
URL: https://github.com/apache/iceberg-python/pull/4436
# Rationale for this change
Bug: appending a fixed-size list of structs fails. The struct branch of
`ArrowProjectionVisitor.list` rebuilds the list from its first two buffers,
which are the validity and offsets buffers only for a variable-size list. A
`FixedSizeListArray` has no offsets buffer, so the rebuilt array is invalid:
```python
import pyarrow as pa
t = pa.table({"v": pa.array([[{"a": 1}, {"a": 2}]],
pa.list_(pa.struct([("a", pa.int32())]), 2))})
catalog.create_table("db.t", schema=t.schema).append(t)
# ArrowInvalid: Non-empty array but offsets are null
```
Fix: rebuild lists and maps around their projected children, preserving
validity and offsets; fixed-size lists retain their list size instead of
offsets. Lists and maps nested in other lists or maps were cast with the
unprojected children, so a renamed struct field (e.g. `a b`) came back as null
on PyArrow 25 and raised on PyArrow 18. A map with list values had the same
gap. This follows #4065's null-list fix; fixed-size lists failed before it too.
Java's lists have no fixed length; Spark writes them through
`SparkParquetWriters`, without an Arrow writer. This fixes PyArrow's shared
read/write projection.
Known limit: PyArrow 18 still rejects some null fixed-size lists in its
Parquet writer, as it does for `fixed_size_list<int>`.
## Are these changes tested?
Yes, `test__to_requested_schema_fixed_size_list_of_structs`,
`test__to_requested_schema_list_of_lists_of_renamed_structs`,
`test__to_requested_schema_map_of_lists_of_renamed_structs`,
`test_append_fixed_size_list_of_structs` and
`test_append_map_of_fixed_size_list_of_structs` in `tests/io/test_pyarrow.py`.
All five fail on main (invalid buffers, or a rename that raises on PyArrow 18
and returns nulls on 25) and pass with this change.
`test_append_map_with_struct_keys` passes on main and guards the map rebuild
for struct keys (apache/arrow#51029).
## Are there any user-facing changes?
Yes, fixed-size lists of structs can be appended, and renamed struct fields
in nested lists and maps are preserved on read and write.
Please add the `changelog` label.
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]