[
https://issues.apache.org/jira/browse/HDDS-16765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16765:
------------------------------
Attachment: MC-3-flag-overwritten-hsync-key-after-commit-validation.patch
> A rejected overwrite commit leaves OVERWRITTEN_HSYNC_KEY on another writer's
> cached open entry
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-16765
> URL: https://issues.apache.org/jira/browse/HDDS-16765
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-3-flag-overwritten-hsync-key-after-commit-validation.patch,
> TestBugMC3FailedOverwriteCommit.java
>
>
> h3. Mechanism
> {{OMKeyCommitRequestWithFSO.validateAndUpdateCache}} flags the hsync writer's
> open entry with {{OVERWRITTEN_HSYNC_KEY}} and puts it in the open file table
> cache ({{:198-213}}) before {{validateAtomicRewrite}} and the bucket quota
> checks. When one of those rejects the commit, nothing undoes the cache put.
> The error response writes nothing to RocksDB, so the flag lives only until
> this OM's double buffer flushes and evicts that epoch. {{OMKeyCommitRequest}}
> ({{:277-291}}) has the same order.
> Each OM flushes and evicts on its own schedule, so the cache content at apply
> time differs between OMs.
> h3. Trigger
> Bucket namespace quota 2, hsync enabled.
> # A creates and hsyncs {{f}}.
> # B creates {{f}} with overwrite.
> # C commits {{g}}.
> # B's close is rejected with {{QUOTA_EXCEEDED}}.
> # A closes while one OM has not yet flushed and evicted B's rejected commit.
> h3. Impact
> The lagging OM rejects A's commit with {{KEY_NOT_FOUND}} while the other OMs
> apply it. {{fileTable}} and {{openFileTable}} differ permanently between OMs,
> and {{getFileStatus}} returns different lengths per OM. If the lagging OM is
> the leader, A's close fails although the overwrite never happened. The same
> happens on a single OM: A's next hsync or close fails with {{KEY_NOT_FOUND}}
> if it is applied before the rejected commit's epoch is evicted.
> h3. Reproduction
> PASS, 3 of 3 runs on a 3 OM mini cluster with one OM's flush held by
> {{OzoneManagerDoubleBuffer#pause}}: [^TestBugMC3FailedOverwriteCommit.java]
> ({{hadoop-ozone/integration-test}}). 40 rounds without the hook did not hit
> the window, so the natural window is narrow on a live OM. An OM that applies
> the two entries back to back, such as a follower catching up or an OM
> replaying its log after a restart, has both in the same unflushed window
> (reasoning from code, not run). A rejection by {{validateAtomicRewrite}} has
> the same effect as the quota rejection.
> h3. Suggested fix
> Move the cache put of the flagged entry below {{validateAtomicRewrite}} and
> the quota checks, next to the other cache updates at the end of
> {{validateAndUpdateCache}}, in both the FSO and the non FSO commit request.
> h3. Patch
> [^MC-3-flag-overwritten-hsync-key-after-commit-validation.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535. It also applies to master at
> c99d291e863.
> In both {{OMKeyCommitRequestWithFSO}} and {{OMKeyCommitRequest}} the flagged
> copy is still built where it was, but the {{addCacheEntry}} call moves into
> the existing "Add to cache of open key table and key table" section, after
> all checks that can reject the commit. Nothing between the old and the new
> position reads that open key entry, so the success path is unchanged.
> Covered by {{testRejectedOverwriteCommitKeepsHsyncedKeyWritable}} in the
> existing {{TestOMKeyCommitRequest}}, inherited by
> {{TestOMKeyCommitRequestWithFSO}}: writer A hsyncs, writer B's overwrite
> commit is rejected with {{QUOTA_EXCEEDED}}, then A's next hsync must return
> {{OK}}. Without the change both variants fail with "expected: <OK> but was:
> <KEY_NOT_FOUND>". The key commit unit suites (48 tests) and {{TestHSync}} (42
> tests) pass, checkstyle is clean. OMs that already diverged are not repaired.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
