Siyao Meng created HDDS-17457:
---------------------------------
Summary: Bucket creation right after a bucket delete is rejected
with TOO_MANY_BUCKETS at the bucket limit, because the deleted bucket is still
counted
Key: HDDS-17457
URL: https://issues.apache.org/jira/browse/HDDS-17457
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-1-count-only-live-buckets-towards-bucket-limit.patch,
TestBugCR1MaxBucketDeleteMarker.java
h3. Mechanism
{{OMBucketCreateRequest.validateMaxBucket}}, called from {{preExecute}} on the
leader, rejects a create when {{bucketTable.getEstimatedKeyCount()}} is at
least {{ozone.om.max.buckets}}. The bucket table has a full cache, so
{{TypedTable.getEstimatedKeyCount}} returns {{FullTableCache.size()}}. A
deleted bucket stays in that cache as an entry with a null value until the
delete is flushed and the cache cleanup evicts it (since 2.0.0 the cleanup runs
every second; in 1.4.x it runs right after the flush, so the window ends at the
flush), and the DeleteBucket reply is sent after apply, before the flush. Until
then the deleted bucket still counts toward the limit.
h3. Trigger
The cluster has {{ozone.om.max.buckets}} buckets. A client deletes one and
creates another within about a second (longer when the double buffer is slow).
No fault injection is needed.
h3. Impact
The create fails with {{TOO_MANY_BUCKETS}} ("Cannot create more than N
buckets") although fewer than N buckets exist. Nothing retries it in the OM,
and a caller that trusts the error believes the limit is reached. A retry after
the eviction succeeds. No state is affected.
h3. Reproduction
PASS on unmodified source, deterministic, about 14 seconds.
[^TestBugCR1MaxBucketDeleteMarker.java] ({{hadoop-ozone/ozone-manager}}) starts
one real OM with single node Ratis ({{OmTestManagers}}) and uses only the
client API with {{ozone.om.max.buckets}} = 3. A create at the limit is rejected
as intended. With the double buffer held by its pause hook, so that the window
does not depend on timing, it deletes one bucket, {{listBuckets}} returns 2
buckets, and the next create is rejected with {{TOO_MANY_BUCKETS}}. After the
delete is flushed and evicted, the same create succeeds. A passing test means
the defect is present.
h3. Patch
[^CR-1-count-only-live-buckets-towards-bucket-limit.patch], against
a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
b6d0181075e.
{{validateMaxBucket}} keeps the cheap count and, only when that count reaches
the limit, counts the live buckets (non-null values) in the bucket table cache,
which holds every bucket. Below the limit nothing changes. At the limit, each
create request walks the in-memory bucket cache once, without taking a lock
(estimated at about 1 ms for 100000 buckets from a pass over a plain skip list
of that size, not measured in an OM). The check runs only on the leader and is
not part of the replicated state, so followers, apply and mixed-version OMs are
not affected. {{FullTableCache.size()}} is unchanged because it also serves the
cache size metric.
Covered by {{preExecuteBucketAfterDeleteAtMaxLimit}} in the existing
{{TestOMBucketCreateRequest}} (also run by
{{TestOMBucketCreateRequestWithFSO}}): at a limit of 1, a create after a delete
must pass, and a further create must still be rejected. Without the change the
first create fails with "Cannot create more than 1 buckets". With it, the
bucket request suites and {{TestBucketManagerImpl}}, {{TestOmMetadataManager}}
and {{TestOzoneManagerRequestHandler}} (161 tests) pass, and checkstyle is
clean. With the patch the reproduction fails, because the create in the window
succeeds.
Found by code review of the OM table cache, double buffer flush and startup
paths, as part of the TLA+ verification effort under HDDS-15926, on commit
a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
