[
https://issues.apache.org/jira/browse/HDDS-9095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17749007#comment-17749007
]
Ivan Andika edited comment on HDDS-9095 at 7/31/23 4:10 AM:
------------------------------------------------------------
Hi [~erose] [~ckj] , related to HDDS-4120, currently OpenKeyCleanupService
deletes all the open keys in the openKeyTable regardless whether the open keys
are MPU-related or not. This might cause orphan MPU in MultipartInfoTable,
since to abort the MultipartInfo, we need to check openKeyTable for MPU-related
open keys, which will fail if the open keys are already deleted. Hence, we
might need to only delete non-MPU open keys in the openKeyCleanupService.
I have raised HDDS-9017 to set isMultipartKey to true for MPU-related open
keys. In this way, MPU open keys can be excluded in getExpiredOpenKeys. The
expired MPU keys can be aborted by a similar background task to abort expired
MPUs (this ticket).
Currently there are two compatibility issues that need to be handled :
# (This can be done as separate ticket) For old MPU open keys that did not set
isMultipartKey to true, how can openKeyCleanupService decides to not delete
these keys . There are few possible solutions
## We repair the OM DB manually (similar to the current effort in HDDS-8097)
to set isMultipartKey to true
## We use regexes for the open MPU DB keys and do not include these keys in
the getExpiredOpenKeys. The patterns are as follows
### Open MPU Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{uploadId}{*})
where uploadId = "UUID.randomUUID().toString() + "-" + UniqueId.next()"
### Open MPU Part Key Info (dbKey =
/\{vol}/\{bucket}/\{key}{*}/\{clientId}{*}) where clientId = UniqueId#next (But
this is also the pattern of normal MPU keys have)
# (This can be handled in this ticket) We might need to deal with the cleaning
up MPUs whose open keys have been deleted by the openKeyCleanupService.
## This might require us to have a new request/response similar to MPU abort
request/response, but without failing when open MPU keys do not exist. This is
not that elegant since it will duplicate most of the MPU abort logic.
Any feedback is appreciated. Thanks.
was (Author: JIRAUSER298977):
Hi [~erose] [~ckj] , related to HDDS-4120, currently OpenKeyCleanupService
deletes all the open keys in the openKeyTable regardless whether the open keys
are MPU-related or not. This might cause orphan MPU in MultipartInfoTable,
since to abort the MultipartInfo, we need to check openKeyTable for MPU-related
open keys, which will fail if the open keys are already deleted. Hence, we
might need to only delete non-MPU open keys in the openKeyCleanupService.
I have raised HDDS-9017 to set isMultipartKey to true for MPU-related open
keys. In this way, MPU open keys can be excluded in getExpiredOpenKeys. The
expired MPU keys can be aborted by a similar background task to abort expired
MPUs (this ticket).
Currently there are two compatibility issues that need to be handled :
# (This can be done as separate ticket) For old MPU open keys that did not set
isMultipartKey to true, how can openKeyCleanupService decides to not delete
these keys . There are few possible solutions
## We repair the OM DB manually (similar to the current effort in HDDS-8097)
to set isMultipartKey to true
## We use regexes for the open MPU DB keys and do not include these keys in
the getExpiredOpenKeys. The patterns are as follows
### Open MPU Key Info (dbKey = /\{vol}/\{bucket}/\{key}{*}/\{uploadId}{*})
where uploadId = "UUID.randomUUID().toString() + "-" + UniqueId.next()"
### Open MPU Part Key Info (dbKey =
/\{vol}/\{bucket}/\{key}{*}/\{clientId}{*}) where clientId = UniqueId#next
# (This can be handled in this ticket) We might need to deal with the cleaning
up MPUs whose open keys have been deleted by the openKeyCleanupService.
## This might require us to have a new request/response similar to MPU abort
request/response, but without failing when open MPU keys do not exist. This is
not that elegant since it will duplicate most of the MPU abort logic.
Any feedback is appreciated. Thanks.
> Implement cleanup service for MultipartInfoTable
> ------------------------------------------------
>
> Key: HDDS-9095
> URL: https://issues.apache.org/jira/browse/HDDS-9095
> Project: Apache Ozone
> Issue Type: Improvement
> Components: OM
> Reporter: Ivan Andika
> Assignee: Ivan Andika
> Priority: Major
> Fix For: 1.4.0
>
>
> In our cluster, there are around few thousands MPU keys in
> multipartInfoTable, with some of them being few months old.
> The reason is that these MPU keys are already initiated, and possibly
> committed few parts, but was not completed / aborted by the user. These
> spaces can be freed.
> Similar to the cleanup service OM open key table (HDDS-4120), we can
> implement clean up on MultipartInfoTable (and related open keys in
> OpenKeyTable) using a background job. However, instead of using a new OM
> request / response, we can reuse the OM MPU abort request / response instead,
> which already handles the cleanup (i.e. expired inflight MPU can be aborted
> after a defined expiry threshold).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
