Siyao Meng created HDDS-17244:
---------------------------------
Summary: Quota repair skips deleted snapshots that are not purged
yet, so the pending deletes they still hold drop out of the bucket's
snapshotUsedBytes
Key: HDDS-17244
URL: https://issues.apache.org/jira/browse/HDDS-17244
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch,
TestBugCR14DeletedSnapshotSkipped.java
h3. Mechanism
{{QuotaRepairTask.recalculateSnapshotDbPendingDeleteQuota}} recomputes
{{snapshotUsedBytes}} and {{snapshotUsedNamespace}} from the active DB
checkpoint and from the snapshot DBs on the repaired buckets' snapshot chains,
but it skips every snapshot whose status is not {{SNAPSHOT_ACTIVE}}
({{loadActiveSnapshot}} on this commit; the patch of HDDS-17241, which lists
the snapshots from the checkpoint's snapshot table instead of the live chain,
keeps the same filter), so deleted snapshots are not scanned.
Deleting a snapshot ({{OMSnapshotDeleteRequest}}) only sets its status to
{{SNAPSHOT_DELETED}}. Its deleted keys, deleted directories and renamed entries
stay in its DB until {{SnapshotDeletingService}} moves them to the next active
snapshot or to the active DB ({{OMSnapshotMoveTableKeysRequest}}, which also
removes them from the deleted snapshot's DB) and then purges the snapshot. None
of these requests changes the bucket's counters: an entry is charged to
{{snapshotUsed*}} from the key delete until {{OMKeyPurgeRequest}} or
{{OMDirectoriesPurgeRequestWithFSO}} purges it. So while a deleted snapshot
still holds entries, they are pending and charged, and nothing else in the
repair counts them. The deleted snapshot's DB is still available
({{OmSnapshotManager.getSnapshot(UUID)}} does not require an active snapshot,
and {{SnapshotDeletingService}} also opens deleted snapshots).
h3. Trigger
* A snapshot of the bucket that was deleted and still holds deleted keys or
directories of the bucket. That lasts until {{SnapshotDeletingService}} has
moved them out, at least one run of the service, longer for large snapshots
(the moves are batched) and while the next snapshot in the chain is also
deleted (the service then skips the snapshot).
* A quota repair of that bucket ({{ozone repair om quota start}}, admin only)
in that window.
h3. Impact
* {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end too low
by what the deleted snapshot still holds (reproduced: 0 and 0 where 3000 and 1
are correct). The value is written to RocksDB and stays until a repair runs
outside the window.
* By the code, not run: when those entries are purged later, the purge
subtracts them again, so the counters stay too low (negative if nothing else is
pending), and the quota checks, which use used plus snapshotUsed, accept writes
beyond the bucket's quota by that amount.
h3. Reproduction
[^TestBugCR14DeletedSnapshotSkipped.java] goes into
{{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
comment) and passes on unmodified source while the bug is present. It deletes a
key, takes a snapshot with the real {{OMSnapshotCreateRequest}} (which moves
the deleted key's row from the active DB into the snapshot DB), deletes the
snapshot with the real {{OMSnapshotDeleteRequest}}, then runs
{{QuotaRepairTask}} and applies the request it submits: the bucket ends at 0
and 0 although the row is still in the deleted snapshot's DB. A control with
the snapshot left active ends at 3000 and 1.
h3. Patch
This patch should land together with or after the one of HDDS-17243. Without
it, the newly scanned deleted snapshots are read in full, every bucket's copies
included: that adds the overcount of HDDS-17243 for the deleted snapshots too,
and a long scan of a snapshot that {{SnapshotDeletingService}} is about to
purge can hold up that purge, which closes the snapshot DB and takes its lock
while the repair still has it open, and with it the OM's double buffer flush
until the scan ends (argued from the code, not run). With the prefix scan of
HDDS-17243 the scan of an emptied deleted snapshot is near instant.
[^CR-14-quota-repair-scans-deleted-snapshots-not-yet-purged.patch] applies on
top of the patch of HDDS-17241 (against
20d95ed37fe35013b5320713665db03c06ee36fa plus that patch); the two also apply
to master at 64d4a791502. It does not conflict with the patch for pending
deletes of other buckets counted from snapshot DBs (filed separately): either
applies on top of the patch of HDDS-17241 alone or together with the other, in
either order.
The status check is removed, so the repair scans every snapshot of the bucket
that is in the checkpoint's snapshot info table, active or deleted. A purged
snapshot is gone from the table and is not scanned, as before; one purged after
the checkpoint fails the run through the check HDDS-17241 adds, like any other
snapshot change during the scan. This does not count an entry twice. A move
takes the entries out of the deleted snapshot's DB when it writes them to the
destination, so a move completed before the repair is counted once, at the
destination. The two writes are separate RocksDB commits, but the move updates
{{lastTransactionInfo}} of both snapshots in the snapshot info cache when it is
applied, so a repair that overlaps a move fails through the check HDDS-17241
adds.
Covered by {{testQuotaRepairCountsDeletedSnapshotsNotYetPurged}} in the
existing {{TestQuotaRepairTask}}: three snapshots, the first two deleted. The
first still holds its key (its next snapshot is deleted too), and the second
has already moved its key to the third, active one with the real
{{OMSnapshotMoveTableKeysRequest}}, so the test checks both the missing count
and that a moved entry is counted once. On top of the HDDS-17241 patch it fails
without the change ("expected: <3000> but was: <2000>"); with it,
{{TestQuotaRepairTask}} and {{TestOMQuotaRepairRequest}} (22 tests) pass,
{{TestOzoneRepairShell}} on a mini cluster passes with both snapshot scan
patches applied, and checkstyle is clean. With the patch the reproduction's
deleted snapshot test fails because the bucket is repaired to 3000 and 1, and
its control passes.
Found during review of a fix for the OM quota repair, as part of the TLA+
verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Related to HDDS-17241, whose fix
changes the same loop. Checked against HDDS issues and apache/ozone pull
requests for duplicates before filing. The attached patch is a proposal for
review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
