[
https://issues.apache.org/jira/browse/HDDS-16785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jheng-Sing Chen reassigned HDDS-16785:
--------------------------------------
Assignee: Jheng-Sing Chen
> Part commit rejected with NO_SUCH_MULTIPART_UPLOAD_ERROR leaves its open key
> behind, so its blocks are queued for deletion twice
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16785
> URL: https://issues.apache.org/jira/browse/HDDS-16785
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Jheng-Sing Chen
> Priority: Minor
> Attachments: MC-5-delete-part-open-key-on-rejected-part-commit.patch,
> TestBugMC5RejectedPartCommitOpenKey.java
>
>
> h3. Mechanism
> When {{S3MultipartUploadCommitPartRequest.validateAndUpdateCache}} finds no
> {{multipartInfoTable}} entry for the upload, it throws
> {{NO_SUCH_MULTIPART_UPLOAD_ERROR}} before any cache update.
> {{S3MultipartUploadCommitPartResponse.checkAndUpdateDB}} then moves the
> blocks named in the commit to {{deletedTable}} but does not delete the open
> key of the part. Part open keys are not flagged as multipart keys, so
> {{OpenKeyCleanupService}} selects the leftover entry after
> {{ozone.om.open.key.expire.threshold}} (default 7d) and
> {{OMOpenKeysDeleteRequest}} puts the same blocks into {{deletedTable}} a
> second time under another key.
> The comment in that response branch ("So, delete this part information.") and
> {{TestS3MultipartUploadCommitPartResponse.testWithMultipartUploadError}},
> which asserts that the open key is gone but never creates it, indicate that
> the entry was meant to be removed.
> h3. Trigger
> # InitiateMultipartUpload, then open a part and write data.
> # The upload is aborted before the part is committed (run with a client
> abort, an abort by {{MultipartUploadCleanupService}} removes the same entry).
> # The part commit is rejected with {{NO_SUCH_MULTIPART_UPLOAD_ERROR}}.
> No fault is needed. A part commit that arrives after CompleteMultipartUpload
> takes the same branch (same code path, no separate reproduction).
> h3. Impact
> * Every rejected part commit leaves an open key entry until the open key
> expiry, and its blocks are then sent to SCM for deletion a second time.
> Datanodes skip duplicate block deletions since HDDS-11267, so no wrong
> datanode state is expected. I ran the OM side only, without datanodes.
> * The leftover open key is still a valid write session. The stock client does
> not use it after the failed part commit, but a {{CommitKey}} on it is
> accepted and publishes a key whose block is already in {{deletedTable}}. The
> block is then sent for deletion while the key stays in {{keyTable}} (run
> against a real OM with a recording SCM client, actual data deletion not run).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC5RejectedPartCommitOpenKey.java] ({{hadoop-ozone/ozone-manager}})
> drives one real OM with its background services through
> {{OzoneManagerProtocol}}, with a recording SCM block client and no datanodes.
> The open key expiry is shortened to 200 ms and the two services are held with
> their own suspend and resume hooks so the tables can be read between the
> steps. The first test shows the same block ID in two {{deletedTable}} entries
> and sent to SCM twice. The second commits the leftover open key as a key and
> shows its block sent for deletion while the key remains. A passing test means
> the defect is present.
> h3. Patch
> [^MC-5-delete-part-open-key-on-rejected-part-commit.patch], against
> d067ba38e7aea917db7efd290b4af53987ec985a. It also applies to master at
> a6332fb5987.
> On the rejected path the request adds a delete marker for the part open key
> to the table cache and the response deletes the key in the batch that writes
> {{deletedTable}}. Because the open key is then gone, the blocks that were
> allocated to the part but not named in the commit are added to the same
> {{deletedTable}} entry. Today only the open key cleanup releases those, so
> without that addition they would leak.
> Covered by {{testValidateAndUpdateCacheMultipartNotFoundDeletesOpenKey}} in
> the existing {{TestS3MultipartUploadCommitPartRequest}} (inherited by the FSO
> subclass), and {{testWithMultipartUploadError}} now creates the open key so
> that its existing assertion is meaningful. Without the change both fail for
> both layouts because the open key is still present. With it the multipart
> request and response unit suites and {{TestOMOpenKeysDeleteRequest}} (142
> tests), the open key cleanup, multipart cleanup and key deleting service
> suites (37 tests) and the multipart tests of {{TestOzoneRpcClient}} on a mini
> cluster (29 tests, including {{testAbortUploadFailWithInProgressPartUpload}}
> and {{testCommitPartAfterCompleteUpload}}) pass, checkstyle is clean. With
> the patch both tests of the reproduction fail, because the open key no longer
> exists after the rejected commit. Entries left behind by earlier rejected
> commits are not touched, the open key cleanup removes them as before.
> Found by TLA+ model checking and code review of the OM expired multipart
> upload cleanup paths under HDDS-15926, on commit
> d067ba38e7aea917db7efd290b4af53987ec985a. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
