[
https://issues.apache.org/jira/browse/HDDS-17049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17049:
------------------------------
Attachment: CR-7-keep-bucket-cache-in-line-with-recreate-and-rebuild.patch
> Recon counts keys of a recreated bucket under the deleted bucket after a
> delete and recreate in one OM flush or behind a rebuild
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17049
> URL: https://issues.apache.org/jira/browse/HDDS-17049
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> CR-7-keep-bucket-cache-in-line-with-recreate-and-rebuild.patch,
> TestBugCR7BucketCacheAfterRecreate.java
>
>
> h3. Mechanism
> Since HDDS-15335 (2.2.0) the OBS and Legacy NSSummary sub-tasks cache each
> bucket's {{OmBucketInfo}}
> ({{NSSummaryTaskDbEventHandler.lookupBucketCached}}) and drop the entry only
> on a bucketTable DELETE event, on the assumption that a recreate is always
> preceded by a delete event. Two paths break that assumption:
> * The OM double buffer writes a flush through one {{RDBBatchOperation}},
> which keeps only the last write per key. A DeleteBucket and a CreateBucket of
> the same name in one flush reach Recon as a single bucketTable UPDATE, so the
> cache keeps the deleted bucket.
> * {{ReconTaskControllerImpl.reInitializeTasks}} rebuilds on a staged task
> instance, swaps the DB in and calls {{init()}} on the live tasks.
> {{NSSummaryTask}} does not override {{init()}}, so the live sub-tasks keep
> their caches. A delete and recreate whose events were never delivered (the
> reason Recon fell back to a rebuild) stays invisible to them.
> In both cases every later key of the recreated bucket is counted under the
> deleted bucket's objectID, while the namespace API ({{BucketEntityHandler}})
> reads the summary of the current objectID.
> h3. Trigger
> * A client deletes a bucket and creates a bucket with the same name, both
> applied in one OM double buffer flush, then writes keys to it.
> * Or a bucket is deleted and recreated while Recon misses the events and
> rebuilds (for example after a full snapshot fallback or an event buffer
> overflow), then keys are written to it.
> Affects releases since 2.2.0, which added the cache.
> h3. Impact
> The namespace summary and disk usage endpoints ({{/namespace/summary}},
> {{/namespace/du}}) under report the recreated bucket, silently. New keys are
> counted under the deleted bucket until Recon restarts or the bucket is
> deleted again, and the keys already miscounted stay missing from the bucket
> until a rebuild recomputes the summaries; a restart does not rebuild
> NSSummary unless a task lags behind the last OM delta. After a restart,
> deleting one of those keys even decrements the new bucket. A rebuild alone
> does not help either, as the second path shows. The reproduction checks the
> summary response; disk usage reads the same NSSummary entry of the bucket's
> current objectID ({{BucketEntityHandler.getDuResponse}}).
> h3. Reproduction
> PASS on unmodified source, deterministic, no fault injection.
> [^TestBugCR7BucketCacheAfterRecreate.java] ({{hadoop-ozone/recon}}) writes
> real OM batches (the bucket changes through {{OMBucketDeleteResponse}} and
> {{OMBucketCreateResponse}}), serves them through {{getUpdatesSince}} and
> applies them to the Recon OM DB as {{OzoneManagerServiceProviderImpl}} does.
> * One flush: Recon receives {{[bucketTable UPDATE /vol/bucket, keyTable PUT
> /vol/bucket/k2]}}. After one more key, the old objectID holds 2 files, the
> new one 0, and the namespace summary reports 0 keys (2 expected).
> * Rebuild: the delete and recreate are applied to the Recon OM DB without
> delivering the events, then the real {{reInitializeTasks}} with the real
> {{ReconDBProvider}} rebuilds (new bucket at 1 file, correct). The next key
> goes to the old objectID; the namespace summary reports 1 key (2 expected).
> A passing test means the defect is present.
> h3. Patch
> [^CR-7-keep-bucket-cache-in-line-with-recreate-and-rebuild.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> Both sub-tasks now keep the cache in line with every bucketTable event: a PUT
> or UPDATE refreshes a cached entry with the event's {{OmBucketInfo}}, a
> DELETE removes it. Buckets that are not cached stay uncached, so the cache
> does not grow, and no extra DB read is needed. Invalidating on every
> bucketTable event was not chosen: every key commit and delete rewrites the
> bucket row, so the cache would be cold in almost every batch, and a missing
> cache entry is one way to reach the NullPointerException of HDDS-17048 (a key
> event of a bucket already deleted in the Recon OM DB). That issue needs its
> own fix and should land with or before this one, since clearing the cache on
> rebuild adds a cold cache window. {{NSSummaryTask.init()}} now clears the
> caches; Recon calls it at startup
> ({{OzoneManagerServiceProviderImpl.start()}}), and the controller after each
> successful rebuild on the event processing thread, between {{process()}}
> calls.
> Covered by {{testBucketCacheInvalidatedOnRecreateInOneFlush}} and
> {{testBucketCacheClearedOnInit}} in the existing
> {{TestNSSummaryTask.TestProcessBucketRecreate}}, which fail without the
> change. With it, the NSSummary task suites, {{TestNSSummaryUnifiedControl}},
> {{TestReconTaskControllerImpl}} and {{TestOMDBUpdatesHandler}} pass (90
> tests, 1 skipped as before) and checkstyle is clean. With the patch both
> reproduction tests fail because the new bucket counts both keys.
> Not fixed: events of one OM flush are grouped per table, and bucketTable
> events come before keyTable events. A key of the old bucket deleted in the
> same flush that recreates the bucket is therefore subtracted from the new
> bucket, which leaves it one key short when a key of the new bucket from that
> flush was counted first (otherwise the delete finds no summary and only logs
> an error). In a run with one flush that deletes the old bucket's last key,
> deletes and recreates the bucket and writes one new key, followed by one more
> key, the new bucket ends at 1 file instead of 2 for three of four key names
> tried, and at 2 for the fourth; before the patch it ends at 0 for all four.
> Before the patch that flush and every later one charged new keys to the old
> bucket instead.
> Found by code review of the Recon namespace summary and task sync paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
