Siyao Meng created HDDS-17458:
---------------------------------
Summary: DeleteBucket after the purge of the bucket's last
snapshot succeeds or fails depending on the OM's flush timing, so OM replicas
diverge
Key: HDDS-17458
URL: https://issues.apache.org/jira/browse/HDDS-17458
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-1-wait-for-snapshot-purge-flush-before-bucket-delete.patch,
TestBugMC1SnapshotBucketDeleteDivergence.java
h3. Mechanism
{{OMBucketDeleteRequest.validateAndUpdateCache}} rejects the delete with
{{CONTAINS_SNAPSHOT}} when {{bucketContainsSnapshot}} finds a snapshot of the
bucket. That check runs on every OM when it applies the log entry, and it ORs
two lookups that both depend on how far this OM has flushed:
* {{bucketContainsSnapshotInCache}} scans the {{snapshotInfoTable}} cache and
returns true for any key of the bucket, including the null value (delete
marker) that {{OMSnapshotPurgeRequest}} puts there. The code carries a TODO for
this case.
* {{bucketContainsSnapshotInTable}} reads RocksDB only, so it still finds a
snapshot whose purge is applied but not yet flushed.
{{snapshotInfoTable}} has a partial cache. The DB row disappears when this OM's
double buffer flushes the purge, and the delete marker shortly after, when the
cache evicts it. Flush timing is local to each OM and is not part of the Ratis
log, and nothing in the apply path waits for a flush. So for the same log
(CreateSnapshot, DeleteSnapshot, SnapshotPurge, DeleteBucket), an OM that has
flushed and evicted the purge deletes the bucket, and an OM that has not
returns {{CONTAINS_SNAPSHOT}} and keeps it.
h3. Trigger
The last snapshot of a bucket is purged by {{SnapshotDeletingService}}, and
DeleteBucket follows while at least one OM has not yet flushed that purge and
another has. A bucket with snapshots cannot be deleted before its snapshots are
purged, so a client that deletes the snapshots and then retries DeleteBucket
until it succeeds sends it right after the purge. An OM that applies the two
entries back to back, for example a follower catching up after a restart or a
slow disk, then decides differently from the leader. No fault injection is
needed.
The reverse order (the leader rejects, a follower that flushed faster deletes)
follows from the same code but was not run. In that case a retry of the client
converges the replicas.
h3. Impact
* Permanent divergence between OM replicas: after both have flushed the
DeleteBucket entry, one has no bucket and the volume's {{usedNamespace}} is
decremented, the other still has the bucket and the old {{usedNamespace}}
(run). Nothing reconciles them short of a checkpoint install.
* After a leader change to the OM that kept it, the bucket that the client was
told is deleted is visible again, and later requests on that bucket name
(create bucket, delete volume) get different results on different OMs (follows
from the state above, not run).
h3. Reproduction
PASS on unmodified source, deterministic, about 3 to 5 seconds.
[^TestBugMC1SnapshotBucketDeleteDivergence.java]
({{hadoop-ozone/ozone-manager}}) builds two OM replicas, each with its own
RocksDB, {{OzoneManagerStateMachine}}, request handler and double buffer (the
{{OzoneManager}} itself is a mock), and applies the same four log entries to
both through {{OzoneManagerStateMachine.applyTransaction}}. Only flush timing
differs, controlled with the double buffer pause hook: r1 flushes the purge and
waits until the cache has evicted it before it applies DeleteBucket, r2 applies
all four entries and then flushes. DeleteBucket returns OK on r1 and
{{CONTAINS_SNAPSHOT}} on r2, and with both flushed to index 4, r1's DB has no
bucket and {{usedNamespace}} 0 while r2's DB has the bucket and
{{usedNamespace}} 1. A passing test means the defect is present. Not exercised:
a real OM HA ring.
h3. Suggested fix
Making the check skip delete markers and read the cache before the DB is not
enough on its own. It lets an OM apply DeleteBucket while the CreateSnapshot of
the same bucket is still in its double buffer, and the flush of that
CreateSnapshot ({{OmSnapshotManager.createOmSnapshotCheckpoint}}) looks up the
bucket ID through the bucket table cache, gets {{BUCKET_NOT_FOUND}} and
terminates the OM. This was observed by running the attached reproduction with
that change applied (on r2). Today the check is protected from this by counting
the delete marker. The patch therefore keeps the delete marker as a reason to
wait instead of a reason to reject.
A deterministic check without a wait is also possible: make the check tombstone
aware with the existing {{OmMetadataManagerImpl.isKeyPresentInTableCache}} and
{{isKeyPresentInTable}} pair (the merge that {{containsIncompleteMPUs}} uses),
and make the CreateSnapshot flush resolve the bucket and volume IDs from the DB
instead of the table cache. At that flush the DB holds exactly the state before
the CreateSnapshot, because
{{OzoneManagerDoubleBuffer.splitReadyBufferAtCreateSnapshot}} puts
CreateSnapshot in a batch of its own after the earlier entries, so the bucket
is there even if a later DeleteBucket has removed it from the cache. The patch
does not take that route because it is a wider change to snapshot code, and
every other bucket lookup at flush time would need the same review.
h3. Patch
[^MC-1-wait-for-snapshot-purge-flush-before-bucket-delete.patch], against
a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
b6d0181075e.
{{bucketContainsSnapshotInCache}} now returns true only for a live snapshot
entry. If it finds only purged entries of the bucket, it waits for this OM's
double buffer to flush ({{OzoneManager.awaitDoubleBufferFlush}}) before the DB
lookup. After that wait the purge and every earlier CreateSnapshot of the
bucket are in the DB on every OM, so the result depends only on the log, and
the bucket is deleted only after its snapshot entries are flushed. The wait
happens only for a DeleteBucket that finds a purged snapshot entry of its own
bucket still in the cache, which is the window this issue is about. It holds
the apply thread, and so every later write on this OM, together with the bucket
lock, until this OM's double buffer has completed its current and its next
flush. That is normally milliseconds, and up to about one second when the flush
thread is idle. It is longer when that flush itself waits, for example for
readers of the purged snapshot to close it (the purge's flush takes the
snapshot DB lock to delete its checkpoint), for a CreateSnapshot checkpoint, or
for the snapshot DB content lock held by snapshot defragmentation. Live
snapshots still reject the delete as before.
Covered by {{testBucketDeleteAfterSnapshotsPurged}} in the existing
{{TestOMBucketDeleteRequest}} (also run by
{{TestOMBucketDeleteRequestWithFSO}}), for a purge that is flushed and one that
is not. Without the change both cases fail with {{expected: <OK> but was:
<CONTAINS_SNAPSHOT>}}. With it, the bucket request, snapshot request, state
machine and double buffer unit suites (254 tests) and {{TestOzoneFsSnapshot}}
on a mini cluster (20 tests, including the rejected delete of a bucket with a
snapshot) pass, and checkstyle is clean. With the patch the reproduction fails
at the assertion on r2's DeleteBucket status, because r2 waits for its flush
and then deletes the bucket as r1 does.
Open pull request apache/ozone#10965 (HDDS-16060) also edits
{{bucketContainsSnapshot}} (it makes the method static and documents that a
deleted but not yet purged snapshot still counts). The two patches conflict
textually there, so whichever lands second needs a rebase. They agree on the
behaviour: this patch still counts such a snapshot.
Found by TLA+ model checking and code review of the OM table cache, double
buffer flush and startup paths under HDDS-15926, on commit
a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
