rdblue commented on a change in pull request #207: Add external schema mappings
for files written with name-based schemas #40
URL: https://github.com/apache/incubator-iceberg/pull/207#discussion_r330279824
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -58,21 +66,19 @@ public Schema record(Schema record, List<String> names,
List<Schema> fields) {
// case where the converted field is non-null is when a map or list is
// selected by lower IDs.
if (selectedIds.contains(fieldId)) {
- filteredFields.add(copyField(field, field.schema()));
+ filteredFields.add(copyField(field, field.schema(), fieldId));
} else if (fieldSchema != null) {
- hasChange = true;
- filteredFields.add(copyField(field, fieldSchema));
+ filteredFields.add(copyField(field, fieldSchema, fieldId));
}
}
- if (hasChange) {
+ if (filteredFields.size() > 0) {
Review comment:
This shouldn't remove the `hasChange` flag. The purpose of `hasChange` is to
signal when this is using the projection of a field. That way, if there are no
known changed fields and all fields were projected, we use the original record.
That's better for caching.
Since you now need to detect the case where the field ID is missing so that
that record gets recreated, you should add a case above that detects when it
isn't there and sets `hasChange = true`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
