ConeyLiu commented on code in PR #4577:
URL: https://github.com/apache/iceberg/pull/4577#discussion_r851855315
##########
core/src/main/java/org/apache/iceberg/avro/BuildAvroProjection.java:
##########
@@ -106,10 +106,16 @@ public Schema record(Schema record, List<String> names,
Iterable<Schema.Field> s
"Missing required field: %s", field.name());
// Create a field that will be defaulted to null. We assign a unique
suffix to the field
// to make sure that even if records in the file have the field it is
not projected.
+ String origFieldName = field.name();
Review Comment:
This problem only happens on metadata table, because we use a common
partition schema for metadata table. For example:
```scala
// create table
spark.sql("create table local.db.test_table(id long, data struct<id: long>)
using iceberg partitioned by (identity(id))").show()
// manifest 0
spark.range(10).map(i => Item(i,
Data(i))).writeTo("local.db.test_table").append()
// update table to add a partiition field on the nested field
spark.sql("alter table local.db.test_table add partition field
identity(data.id)").show()
// manifest 1
spark.range(10).map(i => Item(i,
Data(i))).writeTo("local.db.test_table").append()
spark.sql("select * from local.db.test_table.files").show()
```
The common partition schema is `(Identity(id), Identity(data.id))`. We use
the common schema projecting on the manifest 0 when reading the manifest file,
and we could not find the field(data.id) from the Avro schema which read from
the manifest file. Then we need to create a new Avro field from the missing one
which is why we reached here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org
For additional commands, e-mail: issues-h...@iceberg.apache.org
