rdblue commented on code in PR #18147:
URL: https://github.com/apache/iceberg/pull/18147#discussion_r4031342549
##########
api/src/main/java/org/apache/iceberg/io/CloseableIterable.java:
##########
@@ -132,6 +133,38 @@ protected boolean shouldKeep(E item) {
iterable);
}
+ /**
+ * Filters the given {@link CloseableIterable} and passes each skipped item
to a {@link Consumer}.
+ *
+ * @param skipCallback A consumer used to handle skipped items
+ * @param iterable The underlying {@link CloseableIterable} to filter
+ * @param <E> The underlying type to be iterated
+ * @return A filtered {@link CloseableIterable} that skips items the
predicate does not match
+ */
+ static <E> CloseableIterable<E> filter(
+ Consumer<E> skipCallback, CloseableIterable<E> iterable, Predicate<E>
pred) {
+ Preconditions.checkArgument(null != iterable, "Invalid iterable: null");
+ Preconditions.checkArgument(null != pred, "Invalid predicate: null");
+
+ if (skipCallback != null) {
+ return combine(
+ () ->
+ new FilterIterator<E>(iterable.iterator()) {
+ @Override
+ protected boolean shouldKeep(E item) {
+ boolean matches = pred.test(item);
+ if (!matches) {
+ skipCallback.accept(item);
Review Comment:
This variant of `filter` uses a callback that is passed the item that is
skipped so that we can update the right counter depending on the item. In this
case, update the manifest or data file counter based on the content type.
This iterator is identical to the one removed below, except that it calls
`skipCallback.accept` instead of `counter.increment`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
