[
https://issues.apache.org/jira/browse/HDDS-17458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17458:
------------------------------
Attachment: TestBugMC1SnapshotBucketDeleteDivergence.java
> DeleteBucket after the purge of the bucket's last snapshot succeeds or fails
> depending on the OM's flush timing, so OM replicas diverge
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17458
> URL: https://issues.apache.org/jira/browse/HDDS-17458
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-1-wait-for-snapshot-purge-flush-before-bucket-delete.patch,
> TestBugMC1SnapshotBucketDeleteDivergence.java
>
>
> h3. Mechanism
> {{OMBucketDeleteRequest.validateAndUpdateCache}} rejects the delete with
> {{CONTAINS_SNAPSHOT}} when {{bucketContainsSnapshot}} finds a snapshot of the
> bucket. That check runs on every OM when it applies the log entry, and it ORs
> two lookups that both depend on how far this OM has flushed:
> * {{bucketContainsSnapshotInCache}} scans the {{snapshotInfoTable}} cache and
> returns true for any key of the bucket, including the null value (delete
> marker) that {{OMSnapshotPurgeRequest}} puts there. The code carries a TODO
> for this case.
> * {{bucketContainsSnapshotInTable}} reads RocksDB only, so it still finds a
> snapshot whose purge is applied but not yet flushed.
> {{snapshotInfoTable}} has a partial cache. The DB row disappears when this
> OM's double buffer flushes the purge, and the delete marker shortly after,
> when the cache evicts it. Flush timing is local to each OM and is not part of
> the Ratis log, and nothing in the apply path waits for a flush. So for the
> same log (CreateSnapshot, DeleteSnapshot, SnapshotPurge, DeleteBucket), an OM
> that has flushed and evicted the purge deletes the bucket, and an OM that has
> not returns {{CONTAINS_SNAPSHOT}} and keeps it.
> h3. Trigger
> The last snapshot of a bucket is purged by {{SnapshotDeletingService}}, and
> DeleteBucket follows while at least one OM has not yet flushed that purge and
> another has. A bucket with snapshots cannot be deleted before its snapshots
> are purged, so a client that deletes the snapshots and then retries
> DeleteBucket until it succeeds sends it right after the purge. An OM that
> applies the two entries back to back, for example a follower catching up
> after a restart or a slow disk, then decides differently from the leader. No
> fault injection is needed.
> The reverse order (the leader rejects, a follower that flushed faster
> deletes) follows from the same code but was not run. In that case a retry of
> the client converges the replicas.
> h3. Impact
> * Permanent divergence between OM replicas: after both have flushed the
> DeleteBucket entry, one has no bucket and the volume's {{usedNamespace}} is
> decremented, the other still has the bucket and the old {{usedNamespace}}
> (run). Nothing reconciles them short of a checkpoint install.
> * After a leader change to the OM that kept it, the bucket that the client
> was told is deleted is visible again, and later requests on that bucket name
> (create bucket, delete volume) get different results on different OMs
> (follows from the state above, not run).
> h3. Reproduction
> PASS on unmodified source, deterministic, about 3 to 5 seconds.
> [^TestBugMC1SnapshotBucketDeleteDivergence.java]
> ({{hadoop-ozone/ozone-manager}}) builds two OM replicas, each with its own
> RocksDB, {{OzoneManagerStateMachine}}, request handler and double buffer (the
> {{OzoneManager}} itself is a mock), and applies the same four log entries to
> both through {{OzoneManagerStateMachine.applyTransaction}}. Only flush timing
> differs, controlled with the double buffer pause hook: r1 flushes the purge
> and waits until the cache has evicted it before it applies DeleteBucket, r2
> applies all four entries and then flushes. DeleteBucket returns OK on r1 and
> {{CONTAINS_SNAPSHOT}} on r2, and with both flushed to index 4, r1's DB has no
> bucket and {{usedNamespace}} 0 while r2's DB has the bucket and
> {{usedNamespace}} 1. A passing test means the defect is present. Not
> exercised: a real OM HA ring.
> h3. Suggested fix
> Making the check skip delete markers and read the cache before the DB is not
> enough on its own. It lets an OM apply DeleteBucket while the CreateSnapshot
> of the same bucket is still in its double buffer, and the flush of that
> CreateSnapshot ({{OmSnapshotManager.createOmSnapshotCheckpoint}}) looks up
> the bucket ID through the bucket table cache, gets {{BUCKET_NOT_FOUND}} and
> terminates the OM. This was observed by running the attached reproduction
> with that change applied (on r2). Today the check is protected from this by
> counting the delete marker. The patch therefore keeps the delete marker as a
> reason to wait instead of a reason to reject.
> A deterministic check without a wait is also possible: make the check
> tombstone aware with the existing
> {{OmMetadataManagerImpl.isKeyPresentInTableCache}} and
> {{isKeyPresentInTable}} pair (the merge that {{containsIncompleteMPUs}}
> uses), and make the CreateSnapshot flush resolve the bucket and volume IDs
> from the DB instead of the table cache. At that flush the DB holds exactly
> the state before the CreateSnapshot, because
> {{OzoneManagerDoubleBuffer.splitReadyBufferAtCreateSnapshot}} puts
> CreateSnapshot in a batch of its own after the earlier entries, so the bucket
> is there even if a later DeleteBucket has removed it from the cache. The
> patch does not take that route because it is a wider change to snapshot code,
> and every other bucket lookup at flush time would need the same review.
> h3. Patch
> [^MC-1-wait-for-snapshot-purge-flush-before-bucket-delete.patch], against
> a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
> b6d0181075e.
> {{bucketContainsSnapshotInCache}} now returns true only for a live snapshot
> entry. If it finds only purged entries of the bucket, it waits for this OM's
> double buffer to flush ({{OzoneManager.awaitDoubleBufferFlush}}) before the
> DB lookup. After that wait the purge and every earlier CreateSnapshot of the
> bucket are in the DB on every OM, so the result depends only on the log, and
> the bucket is deleted only after its snapshot entries are flushed. The wait
> happens only for a DeleteBucket that finds a purged snapshot entry of its own
> bucket still in the cache, which is the window this issue is about. It holds
> the apply thread, and so every later write on this OM, together with the
> bucket lock, until this OM's double buffer has completed its current and its
> next flush. That is normally milliseconds, and up to about one second when
> the flush thread is idle. It is longer when that flush itself waits, for
> example for readers of the purged snapshot to close it (the purge's flush
> takes the snapshot DB lock to delete its checkpoint), for a CreateSnapshot
> checkpoint, or for the snapshot DB content lock held by snapshot
> defragmentation. Live snapshots still reject the delete as before.
> Covered by {{testBucketDeleteAfterSnapshotsPurged}} in the existing
> {{TestOMBucketDeleteRequest}} (also run by
> {{TestOMBucketDeleteRequestWithFSO}}), for a purge that is flushed and one
> that is not. Without the change both cases fail with {{expected: <OK> but
> was: <CONTAINS_SNAPSHOT>}}. With it, the bucket request, snapshot request,
> state machine and double buffer unit suites (254 tests) and
> {{TestOzoneFsSnapshot}} on a mini cluster (20 tests, including the rejected
> delete of a bucket with a snapshot) pass, and checkstyle is clean. With the
> patch the reproduction fails at the assertion on r2's DeleteBucket status,
> because r2 waits for its flush and then deletes the bucket as r1 does.
> Open pull request apache/ozone#10965 (HDDS-16060) also edits
> {{bucketContainsSnapshot}} (it makes the method static and documents that a
> deleted but not yet purged snapshot still counts). The two patches conflict
> textually there, so whichever lands second needs a rebase. They agree on the
> behaviour: this patch still counts such a snapshot.
> Found by TLA+ model checking and code review of the OM table cache, double
> buffer flush and startup paths under HDDS-15926, on commit
> a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
