Chi-Hsuan Huang created HDDS-16177:
--------------------------------------
Summary: MPU overwrite on a versioned bucket leaks the previous
key's blocks
Key: HDDS-16177
URL: https://issues.apache.org/jira/browse/HDDS-16177
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Chi-Hsuan Huang
h3. Problem
Completing a multipart upload that overwrites an existing key on a
versioning\-enabled bucket drops the previous key's blocks from OM metadata
without queueing them for deletion. The blocks stay on the datanodes and are
never reclaimed. The previous key's space is also never credited back to the
bucket.
h3. Root cause
In {{S3MultipartUploadCompleteRequest.java:331\-343}}, the previous key\-table
entry is only cleaned up when versioning is off:
{code}
OmKeyInfo keyToDelete =
omMetadataManager.getKeyTable\(getBucketLayout\(\)\).get\(dbOzoneKey\);
boolean isNamespaceUpdate = false;
if \(keyToDelete \!= null && \!omBucketInfo.getIsVersionEnabled\(\)\) {
RepeatedOmKeyInfo oldKeyVersionsToDelete = getOldVersionsToCleanUp\(
keyToDelete, omBucketInfo.getObjectID\(\), trxnLogIndex\);
allKeyInfoToRemove.addAll\(oldKeyVersionsToDelete.getOmKeyInfoList\(\)\);
usedBytesDiff \-= keyToDelete.getReplicatedSize\(\);
} else {
checkBucketQuotaInNamespace\(omBucketInfo, 1L\);
omBucketInfo.incrUsedNamespace\(1L\);
isNamespaceUpdate = true;
}
{code}
{{keyToDelete}} is referenced nowhere else in the request, so on a
versioning\-enabled bucket its blocks never reach {{allKeyInfoToRemove}}, which
is what the response writes to the deleted table.
At the same time the blocks are removed from the metadata. {{getOmKeyInfo}}
starts from the existing key\-table entry and calls
{{omKeyInfo.updateLocationInfoList\(partLocationInfos, true, true\)}}, and
{{OmKeyInfo.updateLocationInfoList}} \({{OmKeyInfo.java:307\-335}}\) performs
{{removeBlocks\(latestVersion\)}} followed by {{addAll\(latestVersion,
updatedBlockLocations\)}}, replacing the latest version group's blocks with the
new parts. MPU completion never calls {{addNewVersion}}, as the comment at
{{S3MultipartUploadCompleteRequest.java:513\-517}} records \("Already a version
exists, so we should add it as a new version. But now as versioning is not
supported, just following the commit key approach"\), so the previous content
is not retained as an older version either.
The result is that the previous key's blocks are referenced by neither the key
table nor the deleted table.
h3. Impact
* Storage on the datanodes is leaked and cannot be reclaimed, since no deletion
is ever scheduled for those blocks.
* The bucket's {{usedBytes}} keeps the previous key's charge, so the counter
stays above what the key table describes.
* Quota repair recounts from the active key table and cannot see the leaked
blocks, so it cannot detect or correct the discrepancy.
* {{usedNamespace}} is also incremented on every such overwrite without a
matching key being added.
h3. Reproduction sketch
On a bucket created with versioning enabled, write a key through a multipart
upload, then overwrite the same key name through a second multipart upload. The
first upload's blocks remain on the datanodes with no metadata referencing them.
h3. Notes
Established by tracing the request path; not yet reproduced end to end. An
integration test using the real client would confirm the blocks are neither in
the deleted table nor referenced by the key table after the second upload.
Found while scoping HDDS\-16127. Pinned source commit 4766aa8609. Analysis
assisted by AI tooling \(Claude Code, Opus 5\).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
