abannon commented on code in PR #19009:
URL: https://github.com/apache/druid/pull/19009#discussion_r2796096638
##########
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java:
##########
@@ -135,6 +131,40 @@ public void kill(List<DataSegment> segments) throws
SegmentLoadingException
}
}
+ /**
+ * Lists all object keys under the given prefix using pagination. S3 returns
at most 1000 keys per request,
+ * so segment directories with more than 1000 files would previously be only
partially deleted. This method
+ * follows continuation tokens to ensure all objects are listed and can be
removed from deep storage.
+ *
+ * @param s3Client client used to communicate with s3
+ * @param s3Bucket the bucket name
+ * @param prefix the key prefix (e.g. segment path ending with /)
+ * @param maxListingLength maximum keys per list request (e.g. 1000)
+ * @return list of all object keys under the prefix
+ */
+ private List<String> listAllKeysWithPrefix(
Review Comment:
Yes, and actually, it might make sense to reuse even more by calling
S3Utils.deleteObjectsInPath()
nevermind, that would disrupt the delete batching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
