clintropolis commented on a change in pull request #9459: Ability to Delete
task logs and segments from S3
URL: https://github.com/apache/druid/pull/9459#discussion_r388691614
##########
File path:
extensions-core/s3-extensions/src/main/java/org/apache/druid/storage/s3/S3DataSegmentKiller.java
##########
@@ -69,8 +85,48 @@ public void kill(DataSegment segment) throws
SegmentLoadingException
}
@Override
- public void killAll()
+ public void killAll() throws IOException
{
- throw new UnsupportedOperationException("not implemented");
+ try {
+ S3Utils.retryS3Operation(
+ () -> {
+ String bucketName = segmentPusherConfig.getBucket();
+ String prefix = segmentPusherConfig.getBaseKey();
+ int maxListingLength = inputDataConfig.getMaxListingLength();
+ ListObjectsV2Result result;
+ String continuationToken = null;
+ do {
+ log.info("Deleting batch of %d segment files from s3 location
[bucket: %s prefix: %s].",
Review comment:
`log.info` is a bit too informative for this operation I think, inside the
loop at least. If you need to log anything, I suggest just counting the number
of keys actually deleted and reporting the total at the end outside of the
loop, or just a single message before the loop happens indicating that a some
deletes are going to happen.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
