Siyao Meng created HDDS-17247:
---------------------------------
Summary: FSO multipart upload completed under a recreated parent
directory builds a new open key instead of reusing the one created at initiate
Key: HDDS-17247
URL: https://issues.apache.org/jira/browse/HDDS-17247
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-8-reuse-initiate-open-key-on-fso-mpu-complete.patch
h3. Mechanism
In an FSO bucket, InitiateMultipartUpload creates the open file entry of the
upload under the object ID of the key's parent directory and records that ID in
{{OmMultipartKeyInfo}}. The entry holds what the client set at initiate, such
as metadata and tags. The parent can be deleted while the upload is open,
because {{OMFileRequest.hasChildren}} only looks at the directory and file
tables.
{{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} then recreates the
missing parents with new object IDs (HDDS-10630). It finds no open entry under
the new parent and builds one from the {{KeyArgs}} of the complete request
instead of reusing the entry created at initiate. Two things follow:
* The completed key takes its metadata and tags from the new entry, so what was
set at initiate is not on the key.
* On success, complete deletes the new entry and the {{multipartInfoTable}}
row, but not the entry under the original parent. Nothing removes it later:
{{OpenKeyCleanupService}} skips open keys flagged as multipart,
{{MultipartUploadCleanupService}} only walks {{multipartInfoTable}},
AbortMultipartUpload returns NO_SUCH_MULTIPART_UPLOAD_ERROR, and bucket delete
does not look at the open file table.
h3. Trigger
# InitiateMultipartUpload for {{dir1/key1}} in an FSO bucket, with metadata or
tags, then upload a part.
# Delete {{dir1}} without the recursive flag. It succeeds, because the open
upload does not count as a child.
# CompleteMultipartUpload succeeds and {{dir1/key1}} is readable.
This is the S3A staging committer flow that HDDS-10630 fixed. No fault or
timing is needed.
h3. Impact
* The completed key has the ETag but not the metadata and tags given at
initiate.
* One open file entry per such upload is left for good. It has no blocks (the
parts move into the completed key), it survives the delete of the key, the
directory and the bucket, and the upload can no longer be aborted.
h3. Reproduction
No standalone reproduction is attached. Two of the regression tests in the
patch show the behaviour: they initiate an upload, commit a part, delete the
parent directory and complete the upload through the request classes, writing
each response to the DB. Without the fix they fail with:
{noformat}
testCompleteAfterParentDirectoryDeletedRemovesOpenKeyOfInitiate:164 expected:
null but was: OmKeyInfo{... keyName='<uuid>/a/b/c/file1', dataSize=0,
keyLocationVersions=[version:0 isMultipartKey:true ], ...}
testCompleteAfterParentDirectoryDeletedKeepsMetadataAndTagsOfInitiate:179
Expecting map: {"ETag"="80404d0c6d24e87f650ff7d1985cd762-1"} to contain
entries: ["content-type"="text/plain"]
{noformat}
I also ran the same steps through {{OzoneManagerProtocol}} on a single OM.
Without the fix, the original entry was still in the open file table after the
bucket was deleted, and the completed key had no metadata or tags from
initiate. With the fix, no entry was left and the key kept both.
h3. Patch
[^CR-8-reuse-initiate-open-key-on-fso-mpu-complete.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
64d4a791502.
When there is no open entry under the recreated parent, complete looks up the
entry created at initiate. Its key uses {{OmMultipartKeyInfo.getParentID}}, the
parent at initiate time, which is the same key the abort fallback from
HDDS-11784 uses. If that entry exists, complete moves it under the new parent
instead of building one from the request. The moved entry also keeps the ACLs
and object ID from initiate, as on a complete whose parent was not deleted. On
success, the original key gets a delete marker in the open file table cache,
and {{S3MultipartUploadCompleteResponseWithFSO}} deletes it in the same batch.
Complete still builds an entry from the request when no entry exists at all, as
before. A complete whose parent still has the object ID it had at initiate
takes none of these branches. Entries left by earlier completes are not touched.
Abort, the expired upload abort and the lifecycle tag lookup still find the
open entry by the current path, so an abort after another request recreated the
parent leaves the original entry. That is a possible follow-up and is not
covered here.
Two alternatives were rejected. Failing the complete would undo HDDS-10630.
Blocking the directory delete would treat an open upload differently from other
open files.
Covered by {{testCompleteAfterParentDirectoryDeletedRemovesOpenKeyOfInitiate}}
and {{testCompleteAfterParentDirectoryDeletedKeepsMetadataAndTagsOfInitiate}}
in the existing {{TestS3MultipartUploadCompleteRequestWithFSO}}, which fail
without the change as shown above.
{{testFailedCompleteAfterParentDirectoryDeletedKeepsOpenKeyOfInitiate}} checks
that a complete that fails after the parent was deleted leaves the original
entry in place, and that a retry then succeeds and removes it. Without the
change, the retry leaves the entry. With it, the multipart request and response
unit suites for both layouts and {{TestOmMetadataManager}} pass (167 tests),
and checkstyle is clean.
Found by code review of the OM volume, bucket and snapshot lifecycle paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
