ivandika3 opened a new pull request, #5325:
URL: https://github.com/apache/ozone/pull/5325
## What changes were proposed in this pull request?
Cleanup service to clean up expired MPUs that are not successfully completed
/ aborted.
Some notable information:
- The code structure is similar to the implementation of the
OpenKeyCleanupService.
- The Request/Response is similar to the
S3MultipartUploadAbortRequest/Response, but unlike normal abort, the request
will not fail if the MPU open key does not exist in the openKeyTable (e.g
previously deleted in the previous openKeyCleanupService)
- The MPU cleanup service has
`ozone.om.open.mpu.parts.cleanup.limit.per.task `which limit the maximum number
of MPU parts to delete rounded up to the nearest MPU (to prevent issues where
MPUs that have parts more than `ozone.om.open.mpu.parts.cleanup.limit.per.task`
never going to be deleted).
- For each run, `Map<OmBucketInfo, List<OmMultipartAbortInfo>>` is created
which contains the batch information to abort the expired MPUs, grouped by
bucket for more efficient locking.
- Logic regarding multipart abort response is refactored to
`AbstractS3MultipartAbortResponse` to accomodate the batched approach of the
cleanup service.
- Since `OmMultipartAbortInfo` contains bucket layout information to
handle cases that MPUs belongs to multiple buckets with different layout,
`AbstractS3MultipartAbortResponse` and the subclasses will get the bucket
layout from `OmMultipartAbortInfo#getBucketLayout` instead of
`OmKeyResponse#getBucketLayout`
- We will skip MPU cleanup if the underlying bucket does not exist
- HDDS-9318 will enforce that bucket cannot be deleted before all the
MPUs under the bucket are completed/aborted/cleaned up
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9194
## How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
