slachiewicz opened a new pull request, #4268: URL: https://github.com/apache/iceberg-python/pull/4268
Closes #4267 # Rationale for this change The spec's column projection rules fill a field that is missing from a data file from the identity partition value in the manifest. `ArrowProjectionVisitor.struct` applied that rule only inside the branch for optional fields and fields with an `initial-default`, so a required identity partition column raised `ResolveError` instead. Tables whose writers keep partition columns out of the data files, such as Hive-style layouts and tables converted from Delta, hit this on every read. A required field now takes the projected value when the partition value is non-null. A null partition value still raises, since it cannot fill a required column. ## Are these changes tested? Yes. `test_identity_transform_column_projection_required_field` covers the projected value, the column's non-nullable type, row filters on the projected column, and the null partition value. It fails without the change. ## Are there any user-facing changes? Scans of such tables return the partition value instead of raising `ResolveError`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
