[
https://issues.apache.org/jira/browse/HDDS-17250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17250:
------------------------------
Affects Version/s: 2.2.1
2.0.0
1.4.0
2.1.2
1.1.0
0.5.0
1.0.0
1.2.1
1.2.0
2.1.0
1.4.1
2.1.1
1.3.0
2.2.0
> Key commit does not check that an object store open key belongs to the
> current bucket of that name
> --------------------------------------------------------------------------------------------------
>
> Key: HDDS-17250
> URL: https://issues.apache.org/jira/browse/HDDS-17250
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 0.5.0, 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0,
> 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: CR-1-reject-commit-of-key-opened-in-earlier-bucket.patch
>
>
> h3. Mechanism
> For OBJECT_STORE and LEGACY buckets the open key table is keyed by
> {{/volume/bucket/key/clientID}}, with no bucket objectID in the key. Bucket
> delete checks the key table ({{OmMetadataManagerImpl.isBucketEmpty}}),
> incomplete multipart uploads and snapshots, but not the open key table, so an
> open key entry can remain after its bucket is deleted.
> {{OMKeyCommitRequest.validateAndUpdateCache}} finds the bucket and the open
> key by name only. Once a bucket with the same name exists again, the commit
> of that entry succeeds and the key is written into the later bucket.
> FSO buckets are not affected: their open key path contains the bucket
> objectID, so the entry is not found under a later bucket.
> h3. Trigger
> An open key of an OBJECT_STORE or LEGACY bucket is committed after the bucket
> was deleted and a bucket with the same name was created in the same volume.
> h3. Impact
> Reproduced on unmodified source through the request classes, with the open
> key entry placed and the bucket table cache entry replaced directly before
> the commit:
> * The commit returns OK and the key appears in the later bucket.
> * The key's size and namespace are charged to the later bucket ({{usedBytes}}
> 0 to 600 with three 200 byte blocks, {{usedNamespace}} 0 to 1).
> h3. Reproduction
> No standalone reproduction is attached.
> {{testValidateAndUpdateCacheKeyOpenedInEarlierBucket}} in the patch runs the
> real {{OMKeyCommitRequest}} {{preExecute}} and {{validateAndUpdateCache}} on
> an OBJECT_STORE bucket against a real {{OmMetadataManagerImpl}}, with object
> IDs in creation order as the OM assigns them, and fails on unmodified source
> with "expected: <KEY_NOT_FOUND> but was: <OK>".
> h3. Patch
> [^CR-1-reject-commit-of-key-opened-in-earlier-bucket.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 64d4a791502.
> Object IDs come from the Ratis transaction index
> ({{OmUtils.getObjectIdFromTxId}}), so a key opened in a bucket always has a
> larger objectID than that bucket.
> {{OMKeyCommitRequest.validateAndUpdateCache}} now rejects the commit with
> {{KEY_NOT_FOUND}} when the open key's objectID is below the bucket's
> ({{Long.compareUnsigned}}, as the epoch sets the top bit), next to the
> existing rejections of deleted or overwritten hsync keys. The same apply path
> serves hsync commits on OBJECT_STORE and LEGACY buckets;
> {{OMKeyCommitRequestWithFSO}} is unchanged, since FSO open keys are keyed by
> bucket objectID. An overwrite keeps the objectID of the key it replaces, so
> the check is skipped when the key table already holds a key with the same
> objectID; keys that were committed this way before the patch therefore stay
> overwritable. No request or proto field is added. Alternatives not taken: an
> open key field naming its bucket (needs a layout gate), or making bucket
> delete refuse while open keys exist, as it does for incomplete multipart
> uploads since HDDS-9318 (blocks deletes until the open keys expire).
> The check changes what an OM applies for such a commit. With the current
> prepare based upgrade, OMs of different versions do not apply the same
> entries; under zero downtime upgrade it would need a version flag once that
> framework is used in OM. The check assumes the transaction index never goes
> back below existing object IDs, which object ID uniqueness already assumes;
> {{ozone repair om update-transaction}} can set a lower index, and until the
> index passes the old values new keys in buckets created above it would be
> rejected at commit. A rejected open key stays in the open key table until
> {{OpenKeyCleanupService}} expires it, as for any other abandoned key.
> The commit test fixtures now give open keys an objectID above their bucket's,
> as the OM does; before, they used 0 while fixture buckets use the current
> time. Covered by three tests in the existing {{TestOMKeyCommitRequest}}:
> {{testValidateAndUpdateCacheKeyOpenedInEarlierBucket}},
> {{testValidateAndUpdateCacheOpenKeyObjectIDBelowBucketAndExistingKey}} (the
> existing key's objectID differs from the open key's, so the overwrite
> exception does not apply and the existing key stays unchanged) and
> {{testValidateAndUpdateCacheOverwriteOfKeyWithLowerObjectID}}. Without the
> change the first two fail with "expected: <KEY_NOT_FOUND> but was: <OK>";
> with the overwrite exception reduced to "no existing key" the second fails
> the same way; without the overwrite exception the third fails with "expected:
> <OK> but was: <KEY_NOT_FOUND>". With it the key request suites pass (448
> tests, including allocate block, open key delete, lease recovery and S3
> commit part), as do {{TestHSync}} and {{TestLeaseRecovery}} (57 tests), and
> checkstyle is clean.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-6682 (checks made in {{preExecute}} that are stale at apply;
> this entry is an open key that outlives its bucket, not one request, and its
> proposed fix (closed PR #3508) records the bucket at the commit's own
> {{preExecute}}, which already sees the later bucket), HDDS-10650 (reported
> that volume, bucket and directory deletes leave open key table entries, for
> hsync files in FSO buckets; its fix covers recursive directory delete only)
> and HDDS-9318 (bucket delete with incomplete multipart uploads). The attached
> patch is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
