[
https://issues.apache.org/jira/browse/HDDS-17247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17247:
------------------------------
Attachment: CR-8-reuse-initiate-open-key-on-fso-mpu-complete.patch
> FSO multipart upload completed under a recreated parent directory builds a
> new open key instead of reusing the one created at initiate
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17247
> URL: https://issues.apache.org/jira/browse/HDDS-17247
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-8-reuse-initiate-open-key-on-fso-mpu-complete.patch
>
>
> h3. Mechanism
> In an FSO bucket, InitiateMultipartUpload creates the open file entry of the
> upload under the object ID of the key's parent directory and records that ID
> in {{OmMultipartKeyInfo}}. The entry holds what the client set at initiate,
> such as metadata and tags. The parent can be deleted while the upload is
> open, because {{OMFileRequest.hasChildren}} only looks at the directory and
> file tables.
> {{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} then recreates
> the missing parents with new object IDs (HDDS-10630). It finds no open entry
> under the new parent and builds one from the {{KeyArgs}} of the complete
> request instead of reusing the entry created at initiate. Two things follow:
> * The completed key takes its metadata and tags from the new entry, so what
> was set at initiate is not on the key.
> * On success, complete deletes the new entry and the {{multipartInfoTable}}
> row, but not the entry under the original parent. Nothing removes it later:
> {{OpenKeyCleanupService}} skips open keys flagged as multipart,
> {{MultipartUploadCleanupService}} only walks {{multipartInfoTable}},
> AbortMultipartUpload returns NO_SUCH_MULTIPART_UPLOAD_ERROR, and bucket
> delete does not look at the open file table.
> h3. Trigger
> # InitiateMultipartUpload for {{dir1/key1}} in an FSO bucket, with metadata
> or tags, then upload a part.
> # Delete {{dir1}} without the recursive flag. It succeeds, because the open
> upload does not count as a child.
> # CompleteMultipartUpload succeeds and {{dir1/key1}} is readable.
> This is the S3A staging committer flow that HDDS-10630 fixed. No fault or
> timing is needed.
> h3. Impact
> * The completed key has the ETag but not the metadata and tags given at
> initiate.
> * One open file entry per such upload is left for good. It has no blocks (the
> parts move into the completed key), it survives the delete of the key, the
> directory and the bucket, and the upload can no longer be aborted.
> h3. Reproduction
> No standalone reproduction is attached. Two of the regression tests in the
> patch show the behaviour: they initiate an upload, commit a part, delete the
> parent directory and complete the upload through the request classes, writing
> each response to the DB. Without the fix they fail with:
> {noformat}
> testCompleteAfterParentDirectoryDeletedRemovesOpenKeyOfInitiate:164 expected:
> null but was: OmKeyInfo{... keyName='<uuid>/a/b/c/file1', dataSize=0,
> keyLocationVersions=[version:0 isMultipartKey:true ], ...}
> testCompleteAfterParentDirectoryDeletedKeepsMetadataAndTagsOfInitiate:179
> Expecting map: {"ETag"="80404d0c6d24e87f650ff7d1985cd762-1"} to contain
> entries: ["content-type"="text/plain"]
> {noformat}
> I also ran the same steps through {{OzoneManagerProtocol}} on a single OM.
> Without the fix, the original entry was still in the open file table after
> the bucket was deleted, and the completed key had no metadata or tags from
> initiate. With the fix, no entry was left and the key kept both.
> h3. Patch
> [^CR-8-reuse-initiate-open-key-on-fso-mpu-complete.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 64d4a791502.
> When there is no open entry under the recreated parent, complete looks up the
> entry created at initiate. Its key uses {{OmMultipartKeyInfo.getParentID}},
> the parent at initiate time, which is the same key the abort fallback from
> HDDS-11784 uses. If that entry exists, complete moves it under the new parent
> instead of building one from the request. The moved entry also keeps the ACLs
> and object ID from initiate, as on a complete whose parent was not deleted.
> On success, the original key gets a delete marker in the open file table
> cache, and {{S3MultipartUploadCompleteResponseWithFSO}} deletes it in the
> same batch. Complete still builds an entry from the request when no entry
> exists at all, as before. A complete whose parent still has the object ID it
> had at initiate takes none of these branches. Entries left by earlier
> completes are not touched.
> Abort, the expired upload abort and the lifecycle tag lookup still find the
> open entry by the current path, so an abort after another request recreated
> the parent leaves the original entry. That is a possible follow-up and is not
> covered here.
> Two alternatives were rejected. Failing the complete would undo HDDS-10630.
> Blocking the directory delete would treat an open upload differently from
> other open files.
> Covered by
> {{testCompleteAfterParentDirectoryDeletedRemovesOpenKeyOfInitiate}} and
> {{testCompleteAfterParentDirectoryDeletedKeepsMetadataAndTagsOfInitiate}} in
> the existing {{TestS3MultipartUploadCompleteRequestWithFSO}}, which fail
> without the change as shown above.
> {{testFailedCompleteAfterParentDirectoryDeletedKeepsOpenKeyOfInitiate}}
> checks that a complete that fails after the parent was deleted leaves the
> original entry in place, and that a retry then succeeds and removes it.
> Without the change, the retry leaves the entry. With it, the multipart
> request and response unit suites for both layouts and
> {{TestOmMetadataManager}} pass (167 tests), and checkstyle is clean.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
