Chu Cheng Li created HDDS-16314:
-----------------------------------
Summary: S3 MPU complete does not filter out blocks still in use
before enqueuing overwritten key versions to deletedTable
Key: HDDS-16314
URL: https://issues.apache.org/jira/browse/HDDS-16314
Project: Apache Ozone
Issue Type: Bug
Components: OM
Reporter: Chu Cheng Li
{{S3MultipartUploadCompleteRequest#validateAndUpdateCache}} enqueues the
overwritten old key versions to deletedTable without calling
{{filterOutBlocksStillInUse}}, unlike {{OMKeyCommitRequest}} (and its FSO
variant), which have applied the filter since HDDS-9146 with an explicit
comment that missing it "causes data loss once those shared blocks are added to
deletedTable and processed by KeyDeletingService for deletion".
If the key being overwritten shares any ContainerBlockID with the newly
completed MPU key, those shared blocks are enqueued to deletedTable and
physically deleted by KeyDeletingService while the live key table entry still
references them.
A concrete route to that state exists through bucket versioning history:
# With bucket versioning enabled, overwriting a key retains the previous
version groups ({{OMKeyRequest#prepareFileInfo}} ->
{{OmKeyInfo#addNewVersion(..., keepOldVersions=true)}}), so the key table entry
accumulates multiple location version groups.
# Bucket versioning is later disabled (supported via
{{OMBucketSetPropertyRequest}}).
# An S3 MPU complete overwrites the key.
{{S3MultipartUploadCompleteRequest#getOmKeyInfo}} builds the new key from the
existing key table entry, and {{updateLocationInfoList(partLocations, true,
true)}} replaces only the latest version's block list, so the older version
groups' blocks remain referenced by the new key. At the same time the overwrite
branch enqueues ALL of keyToDelete's blocks (older version groups included) to
deletedTable.
# KeyDeletingService deletes blocks that the live key table entry still
references -> data loss.
Beyond that route, any state that leaves the key table entry sharing blocks
with the completed MPU key hits the same double-free; the commit path is
already hardened against this class of overlap, the MPU complete path is not.
Not reachable via retried/replayed completes (a duplicate complete fails with
NO_SUCH_MULTIPART_UPLOAD_ERROR because the multipartInfoTable entry is removed
in the same transaction) nor via hsync block overlap (part blocks are freshly
allocated).
Fix: apply the same {{filterOutBlocksStillInUse}} protection in
{{S3MultipartUploadCompleteRequest#validateAndUpdateCache}} before adding the
old key versions to {{allKeyInfoToRemove}}, mirroring {{OMKeyCommitRequest}}.
The shared base class covers both the OBS/LEGACY and FSO paths.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
