Siyao Meng created HDDS-17604:
---------------------------------
Summary: A duplicate snapshot purge removes a new ACTIVE snapshot
that was created with the purged snapshot's name
Key: HDDS-17604
URL: https://issues.apache.org/jira/browse/HDDS-17604
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-1-skip-purge-of-snapshot-not-deleted.patch,
TestBugMC1DuplicatePurgeRecreatedSnapshot.java
h3. Mechanism
A {{SnapshotPurgeRequest}} carries only snapshot table keys
({{/volume/bucket/name}}). {{OMSnapshotPurgeRequest.validateAndUpdateCache}}
purges whatever row the key resolves to when the entry is applied. It does not
check that the row is still the snapshot {{SnapshotDeletingService}} selected,
or that it is still {{SNAPSHOT_DELETED}}. The only guard, added by HDDS-10408,
skips a key whose row is already gone. {{OMSnapshotPurgeResponse.addToDBBatch}}
then reads the row by key from the DB and deletes it together with its
checkpoint directory, also for keys that the request skipped.
So if a second purge for a key is applied after a client created a new snapshot
with the same name, the new ACTIVE snapshot is removed from the snapshot chain,
the snapshot table and the disk on every OM.
h3. Trigger
# Create snapshot {{s1}} in a bucket, then delete it.
# {{SnapshotDeletingService}} on the leader submits a purge for {{s1}}. The
entry commits, then leadership moves to another OM before that OM has applied
it.
# The new leader reports ready as soon as Ratis dispatches its startup entry to
the state machine ({{LeaderStateImpl.checkReady}} from
{{RaftServerImpl.applyLogToStateMachine}}), while OM still has earlier entries
queued on its single apply executor, so the purge can still be queued (the
leader readiness gap is HDDS-17579). {{SnapshotDeletingService}} checks only
{{OzoneManager.isLeaderReady}} and {{shouldIgnoreSnapshot}} (status DELETED and
changes flushed) and keeps no record of submitted purges, so it can build a
second purge for {{s1}}.
# A client creates a snapshot named {{s1}} again, and the create is ordered
after the first purge and before the second. It succeeds with a new snapshot
ID, because the first purge's cache tombstone makes {{isExist}} return false.
# The second purge resolves the key to the new ACTIVE snapshot and purges it.
Steps 2 to 4 need a leader change while the first purge is unapplied on the new
leader, a {{SnapshotDeletingService}} run (default interval 30s) inside that
apply backlog, and a create with the same name submitted in the same window.
The window is narrow. The readiness gap of step 3 is shown on a 3 OM mini
cluster in HDDS-17579; the duplicate purge on top of it was not run on a
cluster. HDDS-16795 is the same pattern in {{KeyDeletingService}}. This issue
is not a duplicate of either: it is a different consequence, and the purge
needs its own check even after HDDS-17579 is fixed.
On 1.4.0 and 1.4.1 the duplicate does not need a leader change:
{{SnapshotDeletingService}} selected snapshots by iterating the on disk
snapshot table, so a run between the first purge's apply and its flush could
build it on a single leader.
h3. Impact
The client's create returned success, but afterwards the snapshot is not found.
Its row is deleted from the snapshot table, its chain entry is removed, so a
chain rebuilt from the table on restart does not have it either, and its
checkpoint directory is deleted. The deep clean flags of the following
snapshots are reset as for a normal purge. Snapshot create moves the bucket's
pending deletes ({{deletedTable}}, {{deletedDirTable}},
{{snapshotRenamedTable}} entries) from the active DB into the new snapshot, and
the purge does not move them on first, so they are lost and their blocks are
never reclaimed (argued, not run). The loss is replicated and permanent.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC1DuplicatePurgeRecreatedSnapshot.java]
({{hadoop-ozone/ozone-manager}}) uses the request level fixture of the snapshot
request tests (real metadata manager, snapshot chain manager, snapshot manager
and checkpoint directories, mocked {{OzoneManager}}). It runs create, delete,
purge, create with the same name and the same purge again through
{{preExecute}}, {{validateAndUpdateCache}} and the response DB update. After
the second purge the new snapshot's row is gone from the cache and the DB, a
chain rebuilt from the table does not have it, its checkpoint directory is
deleted, and {{SnapshotUtils.getSnapshotInfo}} reports it as not found. A
passing test means the defect is present. The test supplies the second purge;
how it arises on a cluster (Trigger, steps 2 and 3) is argued from the code,
not run.
h3. Suggested fix
The patch makes the apply side skip a key whose row is not {{SNAPSHOT_DELETED}}
and passes only the purged keys to the response. It fixes the reported case
without a protocol change. It does not cover a snapshot recreated with the same
name and deleted again before the duplicate purge is applied: that row is
DELETED and is still purged before {{SnapshotDeletingService}} has moved its
pending deletes, which are then lost as above. Carrying the snapshot IDs that
{{SnapshotDeletingService}} selected in {{SnapshotPurgeRequest}} and comparing
them at apply would cover that too, but needs a new request field behind a
layout feature.
h3. Patch
[^MC-1-skip-purge-of-snapshot-not-deleted.patch], against
284fad307bfd129fca3f4966d854ac45ea3b0d11. It also applies to master at
6135e81a09ac6d618dc0ad4d220dd8cf254586b6.
{{OMSnapshotPurgeRequest.validateAndUpdateCache}} skips, with a warning, a key
whose current row is not {{SNAPSHOT_DELETED}}, and gives
{{OMSnapshotPurgeResponse}} only the keys it purged, so the response no longer
deletes the row or checkpoint of a skipped key. A skipped row is not rewritten,
unless the same request updated it as the next snapshot of a purged one.
{{NumSnapshotDeleted}} is decremented by the number of purged keys, which now
are all DELETED.
On a mixed version ring during a rolling upgrade, in exactly the bug case an
upgraded OM keeps the recreated snapshot and an old one purges it; every other
purge behaves the same on both versions.
Covered by two new tests in the existing
{{TestOMSnapshotPurgeRequestAndResponse}}, next to
{{testDuplicateSnapshotPurge}} from HDDS-11997:
{{testDuplicateSnapshotPurgeKeepsRecreatedSnapshot}} (the recreated snapshot is
unchanged in the cache, the DB, the chain and on disk after the duplicate
purge) and {{testSkippedSnapshotNextToPurgedSnapshotIsUpdated}}, run with the
skipped key first and with the purged key first (a skipped snapshot that
follows a purged one keeps its row and checkpoint, and gets its deep clean
flags and previous snapshot IDs updated in the cache and the DB). Without the
change all three cases fail because the recreated snapshot is purged. Four
existing tests in the suite purged ACTIVE snapshots directly; they now mark
them deleted first.
Found by TLA+ model checking and code review of the OM snapshot purge, set
property and double buffer apply paths under HDDS-15926, on commit
284fad307bfd129fca3f4966d854ac45ea3b0d11. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
