rahulsmahadev opened a new pull request, #4436:
URL: https://github.com/apache/iceberg-python/pull/4436

   # Rationale for this change
   
   Bug: appending a fixed-size list of structs fails. The struct branch of 
`ArrowProjectionVisitor.list` rebuilds the list from its first two buffers, 
which are the validity and offsets buffers only for a variable-size list. A 
`FixedSizeListArray` has no offsets buffer, so the rebuilt array is invalid:
   
   ```python
   import pyarrow as pa
   
   t = pa.table({"v": pa.array([[{"a": 1}, {"a": 2}]], 
pa.list_(pa.struct([("a", pa.int32())]), 2))})
   catalog.create_table("db.t", schema=t.schema).append(t)
   # ArrowInvalid: Non-empty array but offsets are null
   ```
   
   Fix: rebuild lists and maps around their projected children, preserving 
validity and offsets; fixed-size lists retain their list size instead of 
offsets. Lists and maps nested in other lists or maps were cast with the 
unprojected children, so a renamed struct field (e.g. `a b`) came back as null 
on PyArrow 25 and raised on PyArrow 18. A map with list values had the same 
gap. This follows #4065's null-list fix; fixed-size lists failed before it too.
   
   Java's lists have no fixed length; Spark writes them through 
`SparkParquetWriters`, without an Arrow writer. This fixes PyArrow's shared 
read/write projection.
   
   Known limit: PyArrow 18 still rejects some null fixed-size lists in its 
Parquet writer, as it does for `fixed_size_list<int>`.
   
   ## Are these changes tested?
   
   Yes, `test__to_requested_schema_fixed_size_list_of_structs`, 
`test__to_requested_schema_list_of_lists_of_renamed_structs`, 
`test__to_requested_schema_map_of_lists_of_renamed_structs`, 
`test_append_fixed_size_list_of_structs` and 
`test_append_map_of_fixed_size_list_of_structs` in `tests/io/test_pyarrow.py`. 
All five fail on main (invalid buffers, or a rename that raises on PyArrow 18 
and returns nulls on 25) and pass with this change. 
`test_append_map_with_struct_keys` passes on main and guards the map rebuild 
for struct keys (apache/arrow#51029).
   
   ## Are there any user-facing changes?
   
   Yes, fixed-size lists of structs can be appended, and renamed struct fields 
in nested lists and maps are preserved on read and write.
   
   Please add the `changelog` label.
   
   This pull request and its description were written by Isaac.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to