rdblue commented on a change in pull request #931: Add residual evaluation for
MR reader
URL: https://github.com/apache/incubator-iceberg/pull/931#discussion_r410479057
##########
File path:
mr/src/main/java/org/apache/iceberg/mr/mapreduce/IcebergInputFormat.java
##########
@@ -259,18 +255,6 @@ public ConfigBuilder skipResidualFiltering() {
return splits;
}
- private static void checkResiduals(CombinedScanTask task) {
- task.files().forEach(fileScanTask -> {
Review comment:
Residual filtering is specific to the object model. That's why the cast to
`Record` works.
I agree with @rdsr that if residuals should be applied (not skipped) then we
need to either ensure that generics are used, or that there are no residuals.
So the check should still happen when `applyResiduals` is true and the
in-memory model is Pig or Hive.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
