[
https://issues.apache.org/jira/browse/HDDS-17241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason O'Sullivan reassigned HDDS-17241:
---------------------------------------
Assignee: Jason O'Sullivan
> Quota repair can leave bucket snapshotUsedBytes negative when a snapshot
> purge commits after the repair's checkpoint and before the snapshot scan
> -------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17241
> URL: https://issues.apache.org/jira/browse/HDDS-17241
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.3.0
> Reporter: Siyao Meng
> Assignee: Jason O'Sullivan
> Priority: Major
> Attachments:
> CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch,
> TestBugCR11QuotaRepairSnapshotPurgeSkew.java
>
>
> h3. Mechanism
> {{QuotaRepairTask.repairTask}} takes a checkpoint of the active OM DB; the
> bucket baselines and all active table counts come from it.
> {{recalculateSnapshotDbPendingDeleteQuota}} instead opens each snapshot DB
> live through {{OmSnapshotManager.getSnapshot}} and counts its current
> {{deletedTable}} and {{deletedDirTable}}. The repair sends the computed value
> minus the baseline for {{snapshotUsedBytes}} and {{snapshotUsedNamespace}},
> and {{OMQuotaRepairRequest.updateBucketInfo}} adds it to the current bucket.
> A request that removes entries from a snapshot DB and lowers the bucket's
> {{snapshotUsed*}} in the same transaction ({{OMKeyPurgeRequest}} or
> {{OMDirectoriesPurgeRequestWithFSO}} with a snapshot table key, as sent by
> the deep clean of the deleting services) and commits after the checkpoint but
> before that snapshot is scanned is then subtracted twice: the scan no longer
> sees the entries, the baseline still has them, and the live bucket has
> already been lowered. A purge applied after both deleted tables of that
> snapshot have been read is harmless. The same undercount happens when a
> snapshot that is active in the checkpoint is deleted, has its entries moved
> to the active DB and is purged before the scan reaches it: the scan walks the
> live snapshot chain, so it skips that snapshot, and the moved entries are in
> neither the checkpoint's active tables nor any scanned snapshot. Nothing
> serializes the repair with the deleting services, and the counters are not
> clamped, so {{OMKeyRequest.checkBucketQuotaInBytes}} and
> {{checkBucketQuotaInNamespace}} (which use used plus snapshotUsed) see the
> bucket that much emptier.
> h3. Trigger
> * A bucket with a snapshot that still holds deleted keys or directories that
> are not yet deep cleaned.
> * A quota repair ({{ozone repair om quota start}}, admin only).
> * A deep clean purge of that snapshot, or its move and purge after a snapshot
> delete, committing between the checkpoint and the scan of that snapshot. The
> window covers opening the checkpoint DB, reading the bucket table and
> scanning the snapshots that come earlier in the scan.
> h3. Impact
> * {{snapshotUsedBytes}} and {{snapshotUsedNamespace}} of the bucket end below
> their true value by what the purge removed, negative if nothing else is
> pending (reproduced: -3000 and -1 where 0 and 0 are correct). The value is
> written to RocksDB and stays until a repair runs without the race.
> * By the code, not run: quota checks undercount the bucket by the same
> amount, so writes beyond the quota are accepted.
> h3. Reproduction
> [^TestBugCR11QuotaRepairSnapshotPurgeSkew.java] goes into
> {{hadoop-ozone/ozone-manager}} (target package and Maven command in the class
> comment) and passes on unmodified source while the bug is present. It deletes
> a key into a snapshot and runs {{QuotaRepairTask}} with the request it
> submits captured and applied. The purge is a real {{OMKeyPurgeRequest}} with
> the snapshot table key; the only control is that it commits on the repair's
> first call to {{getOmSnapshotManager()}}, which comes from the snapshot scan
> after the checkpoint. A control with the purge committed before the repair
> ends at 0 and 0.
> h3. Patch
> [^CR-11-fail-quota-repair-when-snapshot-changes-after-checkpoint.patch],
> against 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master
> at 41d95dd3286.
> The repair now lists the snapshots of each repaired bucket from the
> checkpoint's {{snapshotInfoTable}} instead of the live snapshot chain, so a
> snapshot removed from the chain after the checkpoint is still visited. Every
> request that changes a snapshot DB or removes a snapshot (key and directory
> purge, snapshot moves, snapshot delete and purge) already updates the
> snapshot's {{lastTransactionInfo}} in the snapshot info cache, or removes the
> entry, when it is applied. After scanning a snapshot DB the repair compares
> the live value with the one from the checkpoint and fails the run ("Snapshot
> ... changed after the quota repair checkpoint was taken, run the quota repair
> again") instead of submitting deltas that mix two points in time; a snapshot
> that can no longer be opened fails the run as well. A change applied after
> that comparison cannot have been seen by the scan, and a change flushed
> before the checkpoint is in both, so neither fails the run. The run fails
> whenever a snapshot of a repaired bucket changed between the checkpoint and
> the end of its scan, which includes some changes that would not have skewed
> the result; rerunning is safe. Deep cleaning is on by default, so on a busy
> cluster a long repair can fail several times in a row; {{--buckets}} narrows
> the window to the snapshots of those buckets. Clamping at zero would hide the
> symptom but still be wrong when other deletes are pending. A possible follow
> up is to pause the deleting services for the scan instead, by holding
> {{BOOTSTRAP_LOCK}} from before the checkpoint until the snapshot scan ends,
> together with a double buffer flush, as {{OMDBCheckpointServlet}} does; that
> always converges but stalls deletion and follower bootstrap for the length of
> the scan.
> Covered in the existing {{TestQuotaRepairTask}}:
> {{testQuotaRepairFailsWhenSnapshotPurgedAfterCheckpoint}} fails without the
> change ("expected: 0L but was: -3000L"),
> {{testQuotaRepairFailsWhenSnapshotLeavesChainAfterCheckpoint}} fails without
> it ("expected: 3000L but was: 0L"), and
> {{testQuotaRepairSucceedsWhenSnapshotPurgedBeforeCheckpoint}} checks that a
> purge flushed before the repair does not fail the run. With the change, the
> quota repair, key and directory purge and snapshot move suites (46 tests)
> pass, including the existing {{testQuotaRepairSnapshotDbDeletedTableQuota}}
> for the case without a race, as does {{TestOzoneRepairShell}} on a mini
> cluster, and checkstyle is clean. With the patch the race test of the
> reproduction fails because the repair run fails, and its control passes.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
