[GitHub] [iceberg] shardulm94 commented on a change in pull request #1132: Fix errors when reading options in Avro files with non-null defaults

[GitBox]

shardulm94 commented on a change in pull request #1132:
URL: https://github.com/apache/iceberg/pull/1132#discussion_r444499655



##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -247,8 +255,9 @@ private static Schema copyRecord(Schema record, 
List<Schema.Field> newFields) {
   }
 
   private static Schema.Field copyField(Schema.Field field, Schema newSchema, 
Integer fieldId) {
+    // do not copy over non-null default values as the file is expected to 
have values for fields in the file schema
     Schema.Field copy = new Schema.Field(field.name(),
-        newSchema, field.doc(), field.defaultVal(), field.order());
+        newSchema, field.doc(), hasNonNullDefault(field) ? null : 
field.defaultVal(), field.order());

Review comment:
       The issue is that PruneColumns#copyField can also be called at 
`https://github.com/apache/iceberg/blob/14fb95519b3442092eb6aa02a2608e97e2e8dfd8/core/src/main/java/org/apache/iceberg/avro/PruneColumns.java#L88`.
 In that case, it is possible that `newSchema` is an option schema but with the 
NULL type as the second option (if that is how it is in the file schema) and 
hence `JsonProperties.NULL_VALUE` is not an appropriate default.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org