[
https://issues.apache.org/jira/browse/HDDS-17248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17248:
------------------------------
Attachment: MC-2-keep-bucket-id-of-deleted-keys-moved-from-snapshot.patch
> Deleted keys moved out of a deleted snapshot lose their bucket ID, so their
> purge reduces the usage of a later bucket of the same name
> --------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17248
> URL: https://issues.apache.org/jira/browse/HDDS-17248
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-keep-bucket-id-of-deleted-keys-moved-from-snapshot.patch,
> TestBugMC2SnapshotMoveBucketIdAfterRecreate.java
>
>
> h3. Mechanism
> HDDS-13733 tags each {{deletedTable}} row with the objectID of the bucket the
> key was deleted in, and HDDS-13764 lets {{OMKeyPurgeRequest}} subtract the
> purged size from that bucket's {{snapshotUsedBytes}} and
> {{snapshotUsedNamespace}} only when the tag equals the current bucket's
> objectID. That keeps the purge of a deleted bucket's rows away from a later
> bucket of the same name, except on one path:
> # Bucket delete does not wait for the bucket's {{deletedTable}} rows to be
> purged, so rows of the deleted bucket can still be in the active DB when a
> bucket of the same name is created.
> # Snapshot create on the new bucket moves the active {{deletedTable}} rows by
> the {{/volume/bucket/}} name prefix into the snapshot
> ({{OmSnapshotManager}}), the earlier bucket's rows included, with their tags.
> # When that snapshot is deleted, {{SnapshotDeletingService}} moves the rows
> to the next snapshot or the active DB. {{OMSnapshotMoveTableKeysRequest}}
> passes the current bucket's objectID, and
> {{OMSnapshotMoveTableKeysResponse.addKeysToNextSnapshot}} rebuilds each row
> with {{SnapshotUtils.createMergedRepeatedOmKeyInfoFromDeletedTableEntry}} as
> {{new RepeatedOmKeyInfo(keyInfoList, bucketId)}} when the destination has no
> row of that name. The tag the row carried in the snapshot is dropped.
> # {{KeyDeletingService}} purges the rows, the tag now matches the new bucket,
> and {{purgeSnapshotUsedBytes}} / {{purgeSnapshotUsedNamespace}} subtract
> usage that bucket never had. There is no lower bound.
> Bucket quota checks in {{OMKeyRequest}} use {{getTotalBucketSpace}} and
> {{getTotalBucketNamespace}} (used plus snapshot used), so the negative value
> lets the bucket accept that much more.
> h3. Trigger
> A bucket is deleted while some of its deleted keys are not yet purged (a
> {{KeyDeletingService}} backlog), a bucket of the same name is created, and a
> snapshot of the new bucket is created and later deleted before those rows are
> purged.
> h3. Impact
> Reproduced on unmodified source with three deleted 1 KB keys (replication
> ONE) in an OBJECT_STORE bucket and a new bucket with a namespace quota of 2:
> * The rows come back from the snapshot tagged with the new bucket's objectID
> instead of the deleted bucket's.
> * After the purge the new bucket has {{snapshotUsedBytes=-3072}} and
> {{snapshotUsedNamespace=-3}}.
> * The new bucket then accepted 5 keys under its namespace quota of 2 and
> rejected the 6th ("namespace consumed: 3").
> * Without the snapshot create and delete (same sequence otherwise) the new
> bucket stays at 0.
> The wrong values are persisted and nothing in normal operation corrects them;
> a quota repair recomputes {{snapshotUsed*}} from the tags and should reset
> them (argued, not run). Shown for OBJECT_STORE keys. FSO deleted files are
> keyed by name in {{deletedTable}} and take the same path (not run); FSO
> deleted directories are keyed by object IDs and are not moved by the name
> prefix.
> h3. Reproduction
> [^TestBugMC2SnapshotMoveBucketIdAfterRecreate.java] runs one OM on a mini
> cluster in {{hadoop-ozone/integration-test}} with public client calls only
> (target path and Maven command are in the class comment). The only aid is the
> existing {{KeyDeletingService.suspend()}} / {{resume()}} test hook, which
> stands for the purge backlog.
> {{deletedKeysOfEarlierBucketPassThroughSnapshot}} passes when the bug is
> present; {{deletedKeysOfEarlierBucketWithoutSnapshot}} is the control and
> passes either way.
> h3. Patch
> [^MC-2-keep-bucket-id-of-deleted-keys-moved-from-snapshot.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 64d4a791502.
> {{OMSnapshotMoveTableKeysResponse.addKeysToNextSnapshot}} reads the from
> snapshot's {{deletedTable}} row of each moved key and passes that row's
> bucket ID to {{createMergedRepeatedOmKeyInfoFromDeletedTableEntry}}. Only
> when the row is missing (for example on a replayed flush) does it use the
> current bucket's objectID. Rows deleted in the current bucket keep the same
> tag as today; rows of an earlier bucket keep that bucket's ID, so the
> existing purge check skips them. Rows without a tag keep 0 instead of taking
> the current bucket's ID: they were written before 2.1.0, which added both the
> tag and {{snapshotUsed*}}, so they were never added to {{snapshotUsed*}}, and
> the purge check and quota repair already skip tag 0 in the active DB. The
> legacy {{OMSnapshotMoveDeletedKeysResponse}} re-tags the same way; nothing at
> this commit submits it, and it is not changed. No request or proto change.
> The unused {{bucketId}} field of {{SnapshotMoveKeyInfos}} could carry the tag
> instead, but filling it needs {{KeyManager.getDeletedKeyEntries}} to return
> it, and the response would still have to use it. The cost is one point lookup
> per moved key in the from snapshot's {{deletedTable}}. The patch does not
> repair buckets whose usage is already negative.
> The change alters what an OM writes for such a move. With the current
> upgrade, which goes through OM prepare, OMs of two versions do not apply the
> same entries; under zero downtime upgrade this apply change, like others,
> would need a version flag.
> Covered by the existing
> {{TestOMSnapshotMoveTableKeysResponse.testMoveTableKeysToNextSnapshot}}, now
> run with rows tagged with an earlier bucket's ID and with untagged rows, with
> and without a next snapshot, and asserting the tag of every row at the
> destination, merged rows included. Without the change all four cases fail
> (the moved rows carry the current bucket's ID, for example "expected: <0> but
> was: <1792590778128>"). With it {{TestSnapshotUtils}},
> {{TestOMSnapshotMoveTableKeysResponse}},
> {{TestOMSnapshotMoveTableKeysRequest}},
> {{TestOMSnapshotPurgeRequestAndResponse}},
> {{TestOMKeyPurgeRequestAndResponse}}, {{TestSnapshotDeletingService}} and
> {{TestKeyDeletingService}} pass (107 tests), as does
> {{TestSnapshotDeletingServiceIntegrationTest}}, checkstyle is clean, and the
> attached reproduction no longer shows the bug ({{snapshotUsedNamespace}}
> stays 0).
> Found by TLA+ model checking and code review of the OM volume, bucket and
> snapshot lifecycle paths under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-13733 and HDDS-13764 (added the tag and the purge check this
> path bypasses), HDDS-16126 (fences quota repair deltas by bucket objectID)
> and HDDS-7565 (FSO directory purge of an old bucket updating the new one).
> The attached patch is a proposal for review. Generated with Specula (Claude
> Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
