anoopj commented on code in PR #18816:
URL: https://github.com/apache/iceberg/pull/18816#discussion_r4545376449


##########
core/src/main/java/org/apache/iceberg/V4ManifestReader.java:
##########
@@ -127,12 +144,17 @@ public CloseableIterator<TrackedFile> iterator() {
     }
 
     if (statsFilter != null) {
+      // a manifest reference holds aggregate bounds for only live files, so 
we can't prune them
+      // when reading changes because a non-live entry matching the filter 
could be outside
+      // those aggregate bounds
       files =
           CloseableIterable.filter(
               this::incrementSkipCount,
               files,
-              file -> statsFilter.eval(file.contentStats(), 
file.recordCount()));
-    } else {
+              file ->
+                  (changes && isManifest(file))
+                      || statsFilter.eval(file.contentStats(), 
file.recordCount()));

Review Comment:
   Thanks for the callout. I think we should just drop empty files, since they 
can't be useful for CDF or validations. Made changes accordingly. 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to