bryanck commented on code in PR #7976:
URL: https://github.com/apache/iceberg/pull/7976#discussion_r1287765845
##########
core/src/main/java/org/apache/iceberg/io/ResolvingFileIO.java:
##########
@@ -83,6 +86,29 @@ public void deleteFile(String location) {
io(location).deleteFile(location);
}
+ @Override
+ public void deleteFiles(Iterable<String> pathsToDelete) throws
BulkDeletionFailureException {
+ Map<FileIO, List<String>> pathByFileIO =
+ StreamSupport.stream(pathsToDelete.spliterator(), false)
Review Comment:
Yeah you'd be batching twice, once at this layer, once at the delegate. But
you could set the batch size here to be something very large, just as a sanity
limit to prevent OOMing. I feel even a large fixed size is better than
unlimited, say 100k. Both GCP and AWS have much smaller batch limits so
batching will still be done there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
