[
https://issues.apache.org/jira/browse/HDDS-16177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18106865#comment-18106865
]
Chi-Hsuan Huang commented on HDDS-16177:
----------------------------------------
Not applicable per the discussion on HDDS\-16127. The leak only arises when the
MPU complete path takes the versioning branch, which no write path reaches
today. A non\-versioned overwrite takes the other branch and queues the
previous key's blocks for deletion as expected.
> MPU overwrite on a versioned bucket leaks the previous key's blocks
> -------------------------------------------------------------------
>
> Key: HDDS-16177
> URL: https://issues.apache.org/jira/browse/HDDS-16177
> Project: Apache Ozone
> Issue Type: Bug
> Components: OM
> Reporter: Chi-Hsuan Huang
> Priority: Critical
>
> h3. Problem
> Completing a multipart upload that overwrites an existing key on a
> versioning\-enabled bucket drops the previous key's blocks from OM metadata
> without queueing them for deletion. The blocks stay on the datanodes and are
> never reclaimed. The previous key's space is also never credited back to the
> bucket.
> h3. Root cause
> In {{S3MultipartUploadCompleteRequest.java:331\-343}}, the previous
> key\-table entry is only cleaned up when versioning is off:
> {code}
> OmKeyInfo keyToDelete =
> omMetadataManager.getKeyTable\(getBucketLayout\(\)\).get\(dbOzoneKey\);
> boolean isNamespaceUpdate = false;
> if \(keyToDelete \!= null && \!omBucketInfo.getIsVersionEnabled\(\)\) {
> RepeatedOmKeyInfo oldKeyVersionsToDelete = getOldVersionsToCleanUp\(
> keyToDelete, omBucketInfo.getObjectID\(\), trxnLogIndex\);
> allKeyInfoToRemove.addAll\(oldKeyVersionsToDelete.getOmKeyInfoList\(\)\);
> usedBytesDiff \-= keyToDelete.getReplicatedSize\(\);
> } else {
> checkBucketQuotaInNamespace\(omBucketInfo, 1L\);
> omBucketInfo.incrUsedNamespace\(1L\);
> isNamespaceUpdate = true;
> }
> {code}
> {{keyToDelete}} is referenced nowhere else in the request, so on a
> versioning\-enabled bucket its blocks never reach {{allKeyInfoToRemove}},
> which is what the response writes to the deleted table.
> At the same time the blocks are removed from the metadata. {{getOmKeyInfo}}
> starts from the existing key\-table entry and calls
> {{omKeyInfo.updateLocationInfoList\(partLocationInfos, true, true\)}}, and
> {{OmKeyInfo.updateLocationInfoList}} \({{OmKeyInfo.java:307\-335}}\) performs
> {{removeBlocks\(latestVersion\)}} followed by {{addAll\(latestVersion,
> updatedBlockLocations\)}}, replacing the latest version group's blocks with
> the new parts. MPU completion never calls {{addNewVersion}}, as the comment
> at {{S3MultipartUploadCompleteRequest.java:513\-517}} records \("Already a
> version exists, so we should add it as a new version. But now as versioning
> is not supported, just following the commit key approach"\), so the previous
> content is not retained as an older version either.
> The result is that the previous key's blocks are referenced by neither the
> key table nor the deleted table.
> h3. Impact
> * Storage on the datanodes is leaked and cannot be reclaimed, since no
> deletion is ever scheduled for those blocks.
> * The bucket's {{usedBytes}} keeps the previous key's charge, so the counter
> stays above what the key table describes.
> * Quota repair recounts from the active key table and cannot see the leaked
> blocks, so it cannot detect or correct the discrepancy.
> * {{usedNamespace}} is also incremented on every such overwrite without a
> matching key being added.
> h3. Reproduction sketch
> On a bucket created with versioning enabled, write a key through a multipart
> upload, then overwrite the same key name through a second multipart upload.
> The first upload's blocks remain on the datanodes with no metadata
> referencing them.
> h3. Notes
> Established by tracing the request path; not yet reproduced end to end. An
> integration test using the real client would confirm the blocks are neither
> in the deleted table nor referenced by the key table after the second upload.
> Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
> assisted by AI tooling \(Claude Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
