[
https://issues.apache.org/jira/browse/HDDS-17048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17048:
------------------------------
Attachment: MC-5-skip-key-events-of-deleted-bucket.patch
> Recon NSSummary task fails with NullPointerException on a key event whose
> bucket is already deleted in the Recon OM DB, forcing a full rebuild
> ----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17048
> URL: https://issues.apache.org/jira/browse/HDDS-17048
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-5-skip-key-events-of-deleted-bucket.patch,
> TestBugMC5DeletedBucketKeyEvent.java
>
>
> h3. Mechanism
> Recon commits each fetched OM delta to its OM DB and then buffers the events
> for asynchronous processing, so the Recon OM DB can be ahead of the batch
> being processed. {{NSSummaryTaskWithOBS.processWithOBS}} and
> {{NSSummaryTaskWithLegacy.isBucketLayoutValid}} resolve the bucket of every
> keyTable event with {{NSSummaryTaskDbEventHandler.lookupBucketCached}}, which
> falls back to that DB when the bucket is not cached, and then call
> {{getBucketLayout()}} on the result without a null check. When the bucket was
> deleted in the same delta, or in a later one applied before the batch is
> processed, the lookup returns null and both sub-tasks throw a
> NullPointerException. The reprocess path
> ({{NSSummaryTaskWithOBS.processKeyTableInParallel}}) already skips a null
> bucket.
> A cached bucket does not avoid it. Events of one OM double buffer flush are
> grouped per table, and bucketTable events come before keyTable events. When
> the last key delete and the bucket delete are in one flush, Recon gets the
> bucketTable DELETE first, which drops the cached entry, and the key event
> then looks the bucket up in the DB.
> The NPE fails the whole batch, the controller's retry fails the same way, and
> the controller sets its task failure flag. The OM sync loop then queues a
> {{TASK_FAILURES}} reinitialization, a full rebuild of every Recon OM task.
> Until the rebuild is queued {{consumeOMEvents}} drops new event batches, and
> later batches wait behind the rebuild. The rebuild produces correct data, so
> nothing stays wrong; the cost is the full rebuild and a stale window for all
> Recon OM tasks.
> h3. Trigger
> A key event of an OBS or LEGACY bucket is processed after the bucket is
> deleted in the Recon OM DB, for example:
> # A client deletes the last key of a bucket and then the bucket, and both are
> applied in one OM double buffer flush, which back to back requests such as
> {{ozone sh bucket delete -r}} or {{aws s3 rb --force}} can do. This fails
> whether or not the bucket was cached.
> # Or the key events and the bucket delete are in different flushes, the
> bucket delete reaches the Recon OM DB before the key events are processed
> (the default fetch interval is 5s, longer when event processing lags), and
> the sub-tasks have not cached the bucket, for example a bucket created in the
> same window or one with no key activity since Recon started.
> The unchecked {{getBucketLayout()}} call exists since 1.3.0 for LEGACY
> buckets (HDDS-7121) and since 1.4.1 for OBS buckets (HDDS-7810); those
> versions looked the bucket up for every event without a cache. The bucket
> cache added in 2.2.0 (HDDS-15335) did not change it.
> h3. Impact
> * Every occurrence forces a full rebuild of all Recon OM tasks and leaves
> their data stale until it completes. The other events of the failing batch
> (for other buckets) are applied only by that rebuild. On a large namespace
> the rebuild is long, and frequent bucket cleanup can keep Recon rebuilding.
> * No lasting wrong data is expected, since the rebuild recomputes the summary
> from the Recon OM DB (argued from the code; the attached test stops at the
> task failure flag).
> h3. Reproduction
> PASS on unmodified source, deterministic, no fault injection.
> [^TestBugMC5DeletedBucketKeyEvent.java] ({{hadoop-ozone/recon}}) writes real
> OM DB batches (a key in a live bucket, delete of the last key of a second
> bucket, delete of that bucket), serves them through
> {{RDBStore.getUpdatesSince}}, applies them to the Recon OM DB as
> {{OzoneManagerServiceProviderImpl}} does and hands the events to a real
> {{ReconTaskControllerImpl}} running {{NSSummaryTask}}. Only the task status
> bookkeeping is mocked. It shows {{hasTasksFailed() = true}}, the live bucket
> still at 1 file instead of 2, and the {{getBucketLayout()}} NPE from both
> sub-tasks on process and retry. A passing test means the defect is present.
> h3. Patch
> [^MC-5-skip-key-events-of-deleted-bucket.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> A key event whose bucket is no longer in the Recon OM DB is skipped in both
> sub-tasks, as the reprocess path already does. The bucket and its NSSummary
> entry are unreachable after the delete, so nothing is lost. Covered by
> {{TestNSSummaryTask.TestProcessParallelMixedBatch#testKeyEventOfDeletedBucketDoesNotFailBatch}}
> (bucket not cached) and
> {{#testKeyEventOfDeletedCachedBucketDoesNotFailBatch}} (cached bucket,
> bucketTable DELETE and keyTable DELETE in one batch), which both fail without
> the change. With it, the NSSummary task suites pass (54 tests) and checkstyle
> is clean. With the patch the reproduction fails because the batch succeeds
> and the live bucket counts both keys.
> Related: HDDS-10855 covered the same root cause for the Legacy parent
> directory lookup ({{setKeyParentID}}) and only changed its messages; that
> path still fails a batch when a parent directory key is deleted in the same
> delta and is not changed here.
> Found by TLA+ model checking and code review of the Recon namespace summary
> and task sync paths under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
