RussellSpitzer commented on a change in pull request #2953:
URL: https://github.com/apache/iceberg/pull/2953#discussion_r712378689
##########
File path: core/src/main/java/org/apache/iceberg/avro/PruneColumns.java
##########
@@ -81,13 +83,22 @@ public Schema record(Schema record, List<String> names,
List<Schema> fields) {
Schema fieldSchema = fields.get(field.pos());
// All primitives are selected by selecting the field, but map and list
- // types can be selected by projecting the keys, values, or elements.
+ // types can be selected by projecting the keys, values, or elements.
Empty
+ // Structs can be selected by selecting the record itself instead of its
children.
// This creates two conditions where the field should be selected: if the
// id is selected or if the result of the field is non-null. The only
// case where the converted field is non-null is when a map or list is
// selected by lower IDs.
if (selectedIds.contains(fieldId)) {
- filteredFields.add(copyField(field, field.schema(), fieldId));
+ if (fieldSchema != null) {
+ filteredFields.add(copyField(field, fieldSchema, fieldId));
Review comment:
I think we are actually fine here unless every field is selected because
the logic for has change is a bit confusing.
You either
1. Have a change (Make a new record using the filtered fields)
2. Have no change and filtered fields size is the same as the original
number of fields
3. Have no change and filtered field size is not empty (Make a new record
using the filtered fields)
Currently we have tests hitting 1 and 3 but not 2 :/
I'll add the "hasChange" flag
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
