amogh-jahagirdar commented on a change in pull request #4052:
URL: https://github.com/apache/iceberg/pull/4052#discussion_r801182885
##########
File path: aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java
##########
@@ -94,6 +106,53 @@ public void deleteFile(String path) {
client().deleteObject(deleteRequest);
}
+ /**
+ * Performs a batch deletion of the given paths. This implementation does
not require
+ * that the passed in paths refer to objects which are in the same bucket.
+ * @param paths poths to delete
+ */
+ @Override
+ public void deleteFiles(Iterable<String> paths) {
+ SetMultimap<String, String> bucketToObjects =
Multimaps.newSetMultimap(Maps.newHashMap(), Sets::newHashSet);
+ for (String path : paths) {
+ S3URI location = new S3URI(path);
+ String bucket = location.bucket();
+ String objectKey = location.key();
+ Set<String> objectsInBucket = bucketToObjects.get(bucket);
+ if (objectsInBucket.size() == awsProperties.s3DeletionBatchSize()) {
+ deleteObjectsInBucket(bucket, objectsInBucket);
+ bucketToObjects.removeAll(bucket);
+ }
+ bucketToObjects.get(bucket).add(objectKey);
+ }
+ // Delete the remainder
+ bucketToObjects
+ .asMap()
+ .forEach((bucket, objectsToDelete) -> deleteObjectsInBucket(bucket,
objectsToDelete));
+ }
+
+ private void deleteObjectsInBucket(String bucket, Collection<String>
objects) {
Review comment:
We could, the exception message should then encapsulate the failures
that happened across all the buckets which had some failures.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
