slachiewicz commented on PR #3834: URL: https://github.com/apache/iceberg-python/pull/3834#issuecomment-5385564392
Pushed a second commit that removes the `# This can be removed once this has been fixed: https://github.com/apache/arrow/issues/38809` comment above the rebuild, and adds a test for what the rebuild is actually load-bearing for. The comment is wrong twice over. apache/arrow#38809 is fixed as of pyarrow 19.0.0 (apache/arrow#44587 taught the struct cast to fill absent output fields with null), and a cast really does now handle a field superset, reordering, widening and a null list element. But the rebuild still cannot go, because Iceberg renames by field-id while `cast` matches struct fields by name. Dropping the rebuild and letting the trailing `list_array.cast(arrow_field)` do the work silently nulls out any field renamed inside a list element: ``` with rebuild: [[{'after': 1}], [], None, [{'after': 3}]] rebuild removed: [[{'after': None}], [], None, [{'after': None}]] ``` That is `test__to_requested_schema_renamed_field_in_list_of_structs`, which fails without the rebuild. The null-list test alone does not catch it, since on pyarrow 25 the cast preserves the null on its own. *This comment was created with AI assistance.* -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
