qinghui-xu commented on code in PR #16280:
URL: https://github.com/apache/iceberg/pull/16280#discussion_r3402772872
##########
data/src/main/java/org/apache/iceberg/data/DeleteFilter.java:
##########
@@ -73,13 +72,19 @@ protected DeleteFilter(
Schema expectedSchema,
DeleteCounter counter,
boolean needRowPosCol) {
- this(filePath, deletes, tableSchema::findField, expectedSchema, counter,
needRowPosCol);
+ this(
+ filePath,
+ deletes,
+ ids -> TypeUtil.project(tableSchema, ids),
+ expectedSchema,
+ counter,
+ needRowPosCol);
}
protected DeleteFilter(
String filePath,
List<DeleteFile> deletes,
- Function<Integer, Types.NestedField> fieldLookup,
+ Function<Set<Integer>, Schema> missingSchemaResolver,
Review Comment:
I think we (with @pvary @vigneshc-dd) agree that we want to use
`TypeUtil.project` to correctly find the (nested) columns. But there's
something we may improve over your suggestion:
- method signature: in order to avoid major refactoring, we will likely keep
`DeleteFilter.requiredSchema` field which is a `Schema. So it will be handy if
this method returns a `Schema`
- efficiency: `TypeUtil.project` accepts a `Set` of field ids, it would be
better to call it once with all the needed field ids for the sake of efficiency
- I'd also suggest to define it as instance method instead of `static`, so
that it can be overridden eventually in a subclass (as apparently in spark v4.1
we implement a behavior to look up in historical schemas)
@vigneshc-dd What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
