Siyao Meng created HDDS-16763:
---------------------------------
Summary: A failed FSO CompleteMultipartUpload leaves missing
parent directories in the cache, so later acknowledged creates become orphans
Key: HDDS-16763
URL: https://issues.apache.org/jira/browse/HDDS-16763
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-2-withdraw-cache-entries-of-failed-fso-multipart-complete.patch,
TestBugMC2FailedMpuOrphanDir.java
h3. Mechanism
{{S3MultipartUploadCompleteRequest.validateAndUpdateCache}} adds the missing
FSO parent directories
({{S3MultipartUploadCompleteRequestWithFSO.addMissingParentsToCache}}) and a
synthesized open file entry to the table cache before checks that can still
fail ({{NOT_A_FILE}}, {{NO_SUCH_MULTIPART_UPLOAD_ERROR}}, the part list checks,
{{QUOTA_EXCEEDED}} for the key). The error response writes nothing to RocksDB,
and nothing removes those cache entries until the epoch is evicted after flush.
{{OzoneManagerStateMachine.applyTransaction}} does not wait for the double
buffer flush, so a later request can be applied while the failed request's
directories are still visible in the cache.
h3. Trigger
On an FSO bucket, a {{CompleteMultipartUpload}} that fails (for example an
unknown upload ID) for a key whose parent directories do not exist, followed by
a mkdir or key create and commit under those parents, applied before the failed
request's epoch is flushed and evicted.
h3. Impact
The later request is acknowledged OK and persists a directory or file row whose
parent directory never reaches RocksDB. After eviction the acknowledged
directory or key is unreachable ({{FILE_NOT_FOUND}} / {{KEY_NOT_FOUND}}), stays
unreachable after OM restart, and its row and namespace charge remain as an
orphan. The outcome also depends on each OM's flush timing: an OM that applies
the later create after its own eviction creates the parents itself, so
{{directoryTable}} and {{usedNamespace}} can differ between OMs (argued, not
run).
h3. Reproduction
PASS. [^TestBugMC2FailedMpuOrphanDir.java] ({{hadoop-ozone/integration-test}}),
uninstrumented jars, single OM mini cluster. With two racing client threads and
no hooks, 4 of 300 acknowledged mkdirs were lost. Holding the flush with
{{OzoneManagerDoubleBuffer.pause/unpause}} makes it deterministic. The cross OM
consequence is argued, not run.
h3. Suggested fix
Withdraw the directory and open file cache entries when the complete fails,
under the bucket lock, or restructure the method so that it publishes them only
on success. A plain move of {{addMissingParentsToCache}} and
{{addMultiPartToCache}} below the validation does not work, because the method
resolves the key through those cache entries in between (the complete after
parent deletion case of HDDS-11784 would start failing). HDDS-16117 fixed the
bucket quota side of this site only.
h3. Patch
[^MC-2-withdraw-cache-entries-of-failed-fso-multipart-complete.patch], against
20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
9c6bee807c8.
{{addMissingParentsToCache}} returns the directory table keys it added, and the
existing {{catch}} block calls a new hook, {{removeMissingEntriesFromCache}},
which the FSO request implements by replacing those entries and the synthesized
open key entry with entries without a value at the same transaction index,
while the bucket lock is still held. The keys were verified absent under that
lock before they were added, so this restores the state before the request, and
a later request creates the parents itself on every OM. The success path is
untouched. Publishing only on success would be the cleaner rule but needs the
key resolution on the success path reworked; that is a reviewer's call.
Covered by {{testFailedCompleteDoesNotLeaveMissingParentsInCache}} in the
existing {{TestS3MultipartUploadCompleteRequestWithFSO}} and
{{testDirectoryCreatedAfterFailedCompleteIsPersistedWithItsParent}} in the
existing {{TestOzoneClientMultipartUploadWithFSO}} (double buffer pause and
unpause on the shared cluster). Both fail without the change. The multipart
request and response unit suites (127 tests) and all of
{{TestOzoneClientMultipartUploadWithFSO}} (33 tests, including the HDDS-16117
and HDDS-11784 cases) pass, checkstyle is clean.
Found by TLA+ model checking and code review of the OM FSO namespace mutation
paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
