[
https://issues.apache.org/jira/browse/HDDS-16773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16773:
------------------------------
Attachment: CR-8-delete-keys-release-quota-once-for-repeated-key.patch
> DeleteKeys with a key listed twice releases bucket quota twice
> --------------------------------------------------------------
>
> Key: HDDS-16773
> URL: https://issues.apache.org/jira/browse/HDDS-16773
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-8-delete-keys-release-quota-once-for-repeated-key.patch
>
>
> h3. Mechanism
> {{OMKeysDeleteRequest.validateAndUpdateCache}} resolves every listed name
> before any cache tombstone is written and does not remove duplicate names, so
> a repeated name is appended to {{omKeyInfoList}} twice and
> {{markKeysAsDeletedInCache}} counts it twice. For an empty key the namespace
> is released through {{decrUsedNamespace(emptyKeys, false)}}, which has no
> pending delete counterpart and is never corrected by the purge.
> No client, S3 gateway or OM layer removes duplicates from the request. On FSO
> buckets the repeat does not have to be the same string: {{dir/file}},
> {{dir//file}} and {{/dir/file}} resolve to the same entry and are each
> counted.
> h3. Trigger
> One {{DeleteKeys}} request ({{OzoneBucket.deleteKeys}}, or S3 DeleteObjects)
> that names the same existing zero byte key twice.
> h3. Impact
> Bucket {{usedNamespace}} is permanently one too low per extra occurrence and
> can go negative. It survives the background purge and an OM restart, and
> {{checkBucketQuotaInNamespace}} then accepts more keys than
> {{quotaInNamespace}}. The call can be repeated to exceed the namespace quota
> without bound. For non empty keys and FSO directories the release goes
> through the pending delete counters, so used plus pending (what the quota
> check and the client report) stays correct and the bytes quota bypass did not
> reproduce. {{usedBytes}} / {{usedNamespace}} still stay one occurrence too
> low and the pending counters one occurrence too high, because the purge
> subtracts once.
> h3. Reproduction
> PASS, deterministic, client API on a mini cluster: create a zero byte key,
> call {{deleteKeys}} with its name listed twice, read the bucket's
> {{usedNamespace}}.
> h3. Suggested fix
> Skip entries already resolved in the loop in
> {{OMKeysDeleteRequest.validateAndUpdateCache}} (the FSO subclass shares it).
> Skipping by name is not enough on FSO, where different spellings resolve to
> one entry. Buckets already affected need a quota repair.
> h3. Patch
> [^CR-8-delete-keys-release-quota-once-for-repeated-key.patch], against
> 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> The resolve loop keeps a set of entries already accepted and skips a later
> occurrence of the same entry. The identity is the DB key for OBS and LEGACY
> and {{OmKeyInfo.getPath()}} (parent object ID and file name) for FSO. A
> repeated key is treated as deleted by its first occurrence, so the response
> stays {{OK}} as today and only the accounting changes. Reporting the repeat
> as a partial delete instead would be visible to S3 DeleteObjects and
> {{deleteKeys}} callers, so the patch does not do that.
> Covered by {{testDuplicateKeysReleaseQuotaOnce}} in the existing
> {{TestOMKeysDeleteRequest}} and {{TestOMKeysDeleteRequestWithFSO}}, plus
> {{testDuplicateFileReleasesQuotaOnce}} for the different spellings case.
> Without the change they fail with "expected: 10L but was: 20L", "expected: 3L
> but was: 6L" and "expected: 1L but was: 3L". The key delete request and
> response unit suites pass (70 tests), checkstyle is clean. On master at
> c99d291e863 the product code and base test hunks apply; the
> {{TestOMKeysDeleteRequestWithFSO}} hunk needs a mechanical rebase after
> HDDS-16161.
> Found by TLA+ model checking and code review of the OM FSO namespace mutation
> paths under HDDS-15926, on commit 20e6a1c0f6039248373efd3eb1d7f2afcf7f1535.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
