Copilot commented on code in PR #4268:
URL: https://github.com/apache/iceberg-python/pull/4268#discussion_r4448101797
##########
pyiceberg/io/pyarrow.py:
##########
@@ -2131,8 +2131,13 @@ def struct(
array = self._cast_if_needed(field, field_array)
field_arrays.append(array)
fields.append(self._construct_field(field, array.type))
- elif field.optional or field.initial_default is not None:
- # When an optional field is added, or when a required field
with a non-null initial default is added
+ elif (
+ field.optional
+ or field.initial_default is not None
+ or self._projected_missing_fields.get(field.field_id) is not
None
Review Comment:
A required identity-partitioned field can still have a null partition value
when it is nested inside an optional struct: the parent can be null even though
its child is required when the parent exists. If all parent structs in the
batch are null and the child is omitted from the file, this check raises
`ResolveError` for valid rows. Allow the missing child when it is masked by
null parents, while still rejecting nulls for present parents.
##########
pyiceberg/io/pyarrow.py:
##########
@@ -2131,8 +2131,13 @@ def struct(
array = self._cast_if_needed(field, field_array)
field_arrays.append(array)
fields.append(self._construct_field(field, array.type))
- elif field.optional or field.initial_default is not None:
- # When an optional field is added, or when a required field
with a non-null initial default is added
+ elif (
+ field.optional
+ or field.initial_default is not None
+ or self._projected_missing_fields.get(field.field_id) is not
None
Review Comment:
A top-level required identity column with a non-null `initial_default` still
accepts a null partition value: the default admits this branch, which then
projects `None` into a non-nullable field. This contradicts the stated
null-partition behavior. Validate projected nulls before considering defaults,
accounting for null parents of nested fields, and add coverage for the
default-plus-null case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
