shardulm94 commented on a change in pull request #1352:
URL: https://github.com/apache/iceberg/pull/1352#discussion_r475325724
##########
File path: data/src/main/java/org/apache/iceberg/data/GenericReader.java
##########
@@ -114,14 +117,40 @@
return records;
}
- private Schema fileProjection(boolean hasPosDeletes) {
- if (hasPosDeletes) {
- List<Types.NestedField> columns =
Lists.newArrayList(projection.columns());
+ private Schema fileProjection(List<DeleteFile> posDeletes, List<DeleteFile>
eqDeletes) {
+ Set<Integer> requiredIds = Sets.newLinkedHashSet();
+ if (!posDeletes.isEmpty()) {
+ requiredIds.add(MetadataColumns.ROW_POSITION.fieldId());
+ }
+
+ for (DeleteFile eqDelete : eqDeletes) {
+ requiredIds.addAll(eqDelete.equalityFieldIds());
+ }
+
+ Set<Integer> missingIds =
Sets.newLinkedHashSet(Sets.difference(requiredIds,
TypeUtil.getProjectedIds(projection)));
+
+ if (missingIds.isEmpty()) {
+ return projection;
+ }
+
+ // TODO: support adding nested columns. this will currently fail when
finding nested columns to add
+ List<Types.NestedField> columns = Lists.newArrayList(projection.columns());
+ for (int fieldId : missingIds) {
+ if (fieldId == MetadataColumns.ROW_POSITION.fieldId()) {
+ continue; // add _pos at the end
+ }
+
+ Types.NestedField field = tableSchema.asStruct().field(fieldId);
+ Preconditions.checkArgument(field != null, "Cannot find required field
for ID %s", fieldId);
Review comment:
Interesting constraint here that a column cannot be deleted if there are
live equality delete files depending on the column.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
