cryptoe commented on code in PR #13960:
URL: https://github.com/apache/druid/pull/13960#discussion_r1146289129
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3Utils.java:
##########
@@ -243,59 +243,73 @@ public static S3ObjectSummary
getSingleObjectSummary(ServerSideEncryptingAmazonS
* Delete the files from S3 in a specified bucket, matching a specified
prefix and filter
*
* @param s3Client s3 client
- * @param config specifies the configuration to use when finding matching
files in S3 to delete
+ * @param maxListingLength maximum number of keys to fetch and delete at a
time
* @param bucket s3 bucket
* @param prefix the file prefix
* @param filter function which returns true if the prefix file found
should be deleted and false otherwise.
*
- * @throws Exception
+ * @throws Exception in case of errors
*/
+
public static void deleteObjectsInPath(
ServerSideEncryptingAmazonS3 s3Client,
- S3InputDataConfig config,
+ int maxListingLength,
String bucket,
String prefix,
Predicate<S3ObjectSummary> filter
)
throws Exception
{
- final List<DeleteObjectsRequest.KeyVersion> keysToDelete = new
ArrayList<>(config.getMaxListingLength());
+ deleteObjectsInPath(s3Client, maxListingLength, bucket, prefix, filter,
RetryUtils.DEFAULT_MAX_TRIES);
+ }
+
+ public static void deleteObjectsInPath(
+ ServerSideEncryptingAmazonS3 s3Client,
+ int maxListingLength,
+ String bucket,
+ String prefix,
+ Predicate<S3ObjectSummary> filter,
+ int maxRetries
+ )
+ throws Exception
+ {
+ final List<DeleteObjectsRequest.KeyVersion> keysToDelete = new
ArrayList<>(maxListingLength);
final ObjectSummaryIterator iterator = new ObjectSummaryIterator(
s3Client,
ImmutableList.of(new CloudObjectLocation(bucket, prefix).toUri("s3")),
- config.getMaxListingLength()
+ maxListingLength
);
while (iterator.hasNext()) {
final S3ObjectSummary nextObject = iterator.next();
if (filter.apply(nextObject)) {
keysToDelete.add(new
DeleteObjectsRequest.KeyVersion(nextObject.getKey()));
- if (keysToDelete.size() == config.getMaxListingLength()) {
- deleteBucketKeys(s3Client, bucket, keysToDelete);
- log.info("Deleted %d files", keysToDelete.size());
+ if (keysToDelete.size() == maxListingLength) {
+ deleteBucketKeys(s3Client, bucket, keysToDelete, maxRetries);
Review Comment:
I think each s3 remote call should be retried and not the abstractions we
build on top of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
