[
https://issues.apache.org/jira/browse/HDDS-17239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Attila Doroszlai resolved HDDS-17239.
-------------------------------------
Fix Version/s: 2.3.0
Resolution: Fixed
> Bucket create takes usedBytes, usedNamespace and the snapshot usage counters
> from the request instead of initializing them to 0
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17239
> URL: https://issues.apache.org/jira/browse/HDDS-17239
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0, 1.4.1, 2.1.0,
> 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Assignee: Dian-Xuan Yang
> Priority: Major
> Labels: pull-request-available
> Fix For: 2.3.0
>
> Attachments: CR-12-initialize-bucket-usage-counters-on-create.patch
>
>
> h3. Mechanism
> {{usedBytes}}, {{usedNamespace}}, {{snapshotUsedBytes}} and
> {{snapshotUsedNamespace}} of a bucket are maintained by OM as keys and
> directories are written, deleted and purged. On create,
> {{OMBucketCreateRequest.preExecute}} sets the creation and modification time,
> resolves the encryption key and validates link and replication settings, but
> leaves the four counters of the request's {{BucketInfo}} as they are.
> {{OMBucketCreateRequest.validateAndUpdateCache}} builds the stored bucket
> with {{OmBucketInfo.getFromProtobuf}}, which copies them, and then sets only
> objectID, updateID and the default ACLs. A new bucket therefore starts with
> the counters of the request instead of 0.
> {{RpcClient.createBucket}} never sets these counters, so they are sent as 0
> and buckets created through the Ozone client, {{ozone sh}} and S3 Gateway
> start at 0 today.
> h3. Trigger
> A {{BucketInfo}} that carries any of the four counters reaches
> {{OMBucketCreateRequest.validateAndUpdateCache}}. Nothing on the OM side
> resets them before the bucket is stored.
> h3. Impact
> The bucket table then holds usage that does not match the bucket content, and
> the quota checks of later writes use it. For example, an empty bucket with
> {{quotaInNamespace=1}} created with {{usedNamespace=1}} rejects its first
> directory with {{QUOTA_EXCEEDED}}. Reported usage ({{ozone sh bucket info}})
> is off by the same amount. The values stay until a quota repair recomputes
> them (run on upgrade finalize or by an admin).
> h3. Reproduction
> No standalone reproduction is attached; the regression test in the patch
> shows the behaviour. It runs a CreateBucket request with all four counters
> set through {{preExecute}} and {{validateAndUpdateCache}} and, without the
> change, finds them in the stored bucket for both layouts. The example under
> Impact was checked with a request level test in
> {{hadoop-ozone/ozone-manager}} on unmodified source: a LEGACY bucket with
> {{quotaInNamespace=1}} created with counters 0 accepts one directory from
> {{OMDirectoryCreateRequest}}, and the same bucket created with
> {{usedBytes=1000, usedNamespace=1}} stores both values and rejects it. Not
> run on a cluster.
> h3. Patch
> [^CR-12-initialize-bucket-usage-counters-on-create.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 41d95dd3286.
> {{OMBucketCreateRequest.preExecute}} clears the four counters on the
> {{BucketInfo}} builder it already uses for the creation and modification
> time, so the request written to the Ratis log already carries 0. The apply
> step is unchanged. Requests that already carry 0 behave as before.
> Covered by {{testValidateAndUpdateCacheInitializesUsageCounters}} in the
> existing {{TestOMBucketCreateRequest}} (inherited by
> {{TestOMBucketCreateRequestWithFSO}}), which fails without the change for
> both layouts ({{[1000L, 10L, 100L, 1L]}} stored).
> {{TestBucketManagerImpl.testLinkedBucketResolution}} seeded its source
> bucket's usage through a CreateBucket request; it now writes the source
> bucket to the bucket table directly, so it still checks that the link reports
> the source's usage. With the change the bucket request and response suites,
> {{TestBucketManagerImpl}} and {{TestOmMetadataManager}} pass (126 tests),
> checkstyle is clean.
> {{OMVolumeCreateRequest}} has the same pattern for {{refCount}} and
> {{usedNamespace}} of a volume; that is filed separately with its own patch.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
