rdblue commented on a change in pull request #315: [WIP] Incremental processing
prototype
URL: https://github.com/apache/incubator-iceberg/pull/315#discussion_r361753776
##########
File path: core/src/main/java/org/apache/iceberg/ManifestReader.java
##########
@@ -79,7 +85,12 @@ public static ManifestReader read(InputFile file) {
* @return a manifest reader
*/
public static ManifestReader read(InputFile file, Map<Integer,
PartitionSpec> specsById) {
- return new ManifestReader(file, specsById);
+ return new ManifestReader(file, specsById, Predicates.alwaysTrue());
+ }
+
+ public static ManifestReader read(
+ InputFile file, Map<Integer, PartitionSpec> specsById,
Predicate<ManifestEntry> manifestEntryPredicate) {
Review comment:
I don't think this is the right place to add a predicate. This class is used
to either get the entire list of entries by calling `entries` or to get the
entire list of files by using it as an `Iterator`.
All filtering should be done using the refinement methods, like
`filterPartitions` or `filterRows`. Some classes, like
[`ManifestGroup`](https://github.com/apache/incubator-iceberg/blob/master/core/src/main/java/org/apache/iceberg/ManifestGroup.java#L178-L202)
use the reader and filters to get entries and then add additional predicates.
I think we should take that approach instead of changing how to use
`ManifestReader`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
