Siyao Meng created HDDS-17243:
---------------------------------
Summary: Quota repair counts pending deletes of a bucket again for
every snapshot of another bucket that holds a copy of them
Key: HDDS-17243
URL: https://issues.apache.org/jira/browse/HDDS-17243
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-13-quota-repair-scans-only-own-bucket-rows-of-snapshot-db.patch,
TestBugCR13SnapshotDbCrossBucketCount.java
h3. Mechanism
{{QuotaRepairTask.recalculateSnapshotDbPendingDeleteQuota}} recomputes
{{snapshotUsedBytes}} and {{snapshotUsedNamespace}} from the active DB
checkpoint and from every snapshot DB on the repaired buckets' snapshot chains.
For a snapshot DB it scans the whole {{deletedTable}} and {{deletedDirTable}}
({{recalculateDeletedKeyUsages}}, {{recalculateDeletedDirNamespace}}) and
credits each row to the bucket it belongs to.
A snapshot DB is a checkpoint of the whole OM DB.
{{OmSnapshotManager.createOmSnapshotCheckpoint}} removes only the snapshot's
own bucket prefix from the active {{deletedTable}}, {{deletedDirTable}} and
{{snapshotRenamedTable}}, so the pending deletes of every other bucket stay in
the active DB and are also copied into the snapshot DB. The repair counts them
once from the active DB checkpoint and once more from every snapshot DB of
another bucket that holds a copy. The code that reads or changes a snapshot's
deleted tables ({{SnapshotDeletingService}} through
{{KeyManagerImpl.getDeletedKeyEntries}} and {{getDeletedDirEntries}},
{{OMSnapshotMoveTableKeysRequest}}) always restricts itself to the snapshot's
bucket prefix, so the copies are never removed from the snapshot DB, and the
repair keeps counting them even after the real entries have been purged from
the active DB.
h3. Trigger
* A snapshot of bucket A taken while bucket B (any other bucket in the cluster)
had pending deletes in the active {{deletedTable}} or {{deletedDirTable}}.
* A quota repair that includes both buckets ({{ozone repair om quota start}},
admin only; a repair without {{--buckets}} includes every bucket). The repair
scans only the snapshots of the buckets it repairs, so a repair of bucket B
alone is not affected.
h3. Impact
* {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of bucket B end too high
by one copy of its rows per snapshot DB of another bucket that holds them
(reproduced: 6000 bytes and 2 entries where 3000 and 1 are correct, with one
snapshot of another bucket; 3000 bytes where 1000 is correct, with two). The
value is written to RocksDB, and every later repair computes the same excess
again while those snapshots exist.
* By the code, not run: the quota checks use used plus snapshotUsed, so bucket
B can reject writes with {{QUOTA_EXCEEDED}} before it reaches its quota. On a
real cluster the excess can be smaller than one full copy per snapshot, because
{{SstFilteringService}} drops the SST files of a snapshot DB that hold no key
of the snapshot's own bucket, and with them some of the copies.
h3. Reproduction
[^TestBugCR13SnapshotDbCrossBucketCount.java] goes into
{{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
comment) and passes on unmodified source while the bug is present. It deletes
one key of bucket B, takes a snapshot of bucket A with the real
{{OMSnapshotCreateRequest}}, then runs {{QuotaRepairTask}} and applies the
request it submits: bucket B ends at 6000 bytes and 2 entries. A control
without the snapshot ends at 3000 and 1.
h3. Patch
[^CR-13-quota-repair-scans-only-own-bucket-rows-of-snapshot-db.patch] applies
on top of the patch of HDDS-17241 (against
20d95ed37fe35013b5320713665db03c06ee36fa plus that patch); the two also apply
to master at 64d4a791502. It does not conflict with the patch for deleted
snapshots that are not purged yet (filed separately): either applies on top of
the patch of HDDS-17241 alone or together with the other, in either order.
The snapshot DB scan now iterates only the snapshot's own bucket prefix,
{{/volume/bucket/}} in {{deletedTable}} and {{/volumeId/bucketId/}} in
{{deletedDirTable}}: the same rows that snapshot create removes from the active
DB and that {{SnapshotDeletingService}} moves. Each pending entry is then
counted once, from the active DB or from the one snapshot DB that holds it,
also for a bucket with several snapshots. The scan of the active DB checkpoint
is unchanged, and a snapshot DB scan now reads only its own bucket's rows
instead of every deleted row in the cluster. This also takes
{{SstFilteringService}} out of the picture: it deletes SST files of a snapshot
DB that hold only other buckets' rows without updating the snapshot's
{{lastTransactionInfo}}, so before this change a filter run during the repair
could change what was counted without the check of HDDS-17241 noticing.
Covered by {{testQuotaRepairCountsOnlyOwnBucketRowsOfSnapshotDb}} in the
existing {{TestQuotaRepairTask}}: two FSO buckets with deleted keys and
directories, and two snapshots of one of them. The first bucket's name and ID
are string prefixes of the other bucket's, so the test also shows that the scan
stops at the path separator. On top of the HDDS-17241 patch it fails without
the change ("expected: <1000> but was: <3000>"); with it,
{{TestQuotaRepairTask}} and {{TestOMQuotaRepairRequest}} (22 tests) pass,
{{TestOzoneRepairShell}} on a mini cluster passes with both snapshot scan
patches applied, and checkstyle is clean. With the patch the reproduction's
snapshot test fails because bucket B is repaired to 3000 and 1, and its control
passes.
Found during review of a fix for the OM quota repair, as part of the TLA+
verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Related to HDDS-17241, whose fix
changes the same loop. Checked against HDDS issues and apache/ozone pull
requests for duplicates before filing. The attached patch is a proposal for
review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
