amogh-jahagirdar commented on code in PR #5379:
URL: https://github.com/apache/iceberg/pull/5379#discussion_r933671501
##########
aws/src/main/java/org/apache/iceberg/aws/s3/S3FileIO.java:
##########
@@ -167,6 +168,29 @@ public Map<String, String> properties() {
*/
@Override
public void deleteFiles(Iterable<String> paths) throws
BulkDeletionFailureException {
+ deleteFilesWithBatchSize(paths, awsProperties.s3FileIoDeleteBatchSize());
+ }
+
+ /**
+ * Deletes the given paths in a batched manner with the given executor
service
+ *
+ * @param paths paths to delete
+ * @param svc executor service to use for the deletion
+ */
+ @Override
+ public void deleteFiles(Iterable<String> paths, ExecutorService svc)
+ throws BulkDeletionFailureException {
+ Iterable<List<String>> pathBatches =
+ Iterables.partition(paths, awsProperties.s3FileIoDeleteBatchSize());
+ Tasks.foreach(pathBatches)
+ .noRetry()
Review Comment:
Missed this, but I think the default S3 SDK policy is to retry on
throttling? I will double check that.
would we want to add another layer of retries considering users can pass in
their own S3 client with perhaps their own retry policy?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
