Siyao Meng created HDDS-17048:
---------------------------------
Summary: Recon NSSummary task fails with NullPointerException on a
key event whose bucket is already deleted in the Recon OM DB, forcing a full
rebuild
Key: HDDS-17048
URL: https://issues.apache.org/jira/browse/HDDS-17048
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-5-skip-key-events-of-deleted-bucket.patch,
TestBugMC5DeletedBucketKeyEvent.java
h3. Mechanism
Recon commits each fetched OM delta to its OM DB and then buffers the events
for asynchronous processing, so the Recon OM DB can be ahead of the batch being
processed. {{NSSummaryTaskWithOBS.processWithOBS}} and
{{NSSummaryTaskWithLegacy.isBucketLayoutValid}} resolve the bucket of every
keyTable event with {{NSSummaryTaskDbEventHandler.lookupBucketCached}}, which
falls back to that DB when the bucket is not cached, and then call
{{getBucketLayout()}} on the result without a null check. When the bucket was
deleted in the same delta, or in a later one applied before the batch is
processed, the lookup returns null and both sub-tasks throw a
NullPointerException. The reprocess path
({{NSSummaryTaskWithOBS.processKeyTableInParallel}}) already skips a null
bucket.
A cached bucket does not avoid it. Events of one OM double buffer flush are
grouped per table, and bucketTable events come before keyTable events. When the
last key delete and the bucket delete are in one flush, Recon gets the
bucketTable DELETE first, which drops the cached entry, and the key event then
looks the bucket up in the DB.
The NPE fails the whole batch, the controller's retry fails the same way, and
the controller sets its task failure flag. The OM sync loop then queues a
{{TASK_FAILURES}} reinitialization, a full rebuild of every Recon OM task.
Until the rebuild is queued {{consumeOMEvents}} drops new event batches, and
later batches wait behind the rebuild. The rebuild produces correct data, so
nothing stays wrong; the cost is the full rebuild and a stale window for all
Recon OM tasks.
h3. Trigger
A key event of an OBS or LEGACY bucket is processed after the bucket is deleted
in the Recon OM DB, for example:
# A client deletes the last key of a bucket and then the bucket, and both are
applied in one OM double buffer flush, which back to back requests such as
{{ozone sh bucket delete -r}} or {{aws s3 rb --force}} can do. This fails
whether or not the bucket was cached.
# Or the key events and the bucket delete are in different flushes, the bucket
delete reaches the Recon OM DB before the key events are processed (the default
fetch interval is 5s, longer when event processing lags), and the sub-tasks
have not cached the bucket, for example a bucket created in the same window or
one with no key activity since Recon started.
The unchecked {{getBucketLayout()}} call exists since 1.3.0 for LEGACY buckets
(HDDS-7121) and since 1.4.1 for OBS buckets (HDDS-7810); those versions looked
the bucket up for every event without a cache. The bucket cache added in 2.2.0
(HDDS-15335) did not change it.
h3. Impact
* Every occurrence forces a full rebuild of all Recon OM tasks and leaves their
data stale until it completes. The other events of the failing batch (for other
buckets) are applied only by that rebuild. On a large namespace the rebuild is
long, and frequent bucket cleanup can keep Recon rebuilding.
* No lasting wrong data is expected, since the rebuild recomputes the summary
from the Recon OM DB (argued from the code; the attached test stops at the task
failure flag).
h3. Reproduction
PASS on unmodified source, deterministic, no fault injection.
[^TestBugMC5DeletedBucketKeyEvent.java] ({{hadoop-ozone/recon}}) writes real OM
DB batches (a key in a live bucket, delete of the last key of a second bucket,
delete of that bucket), serves them through {{RDBStore.getUpdatesSince}},
applies them to the Recon OM DB as {{OzoneManagerServiceProviderImpl}} does and
hands the events to a real {{ReconTaskControllerImpl}} running
{{NSSummaryTask}}. Only the task status bookkeeping is mocked. It shows
{{hasTasksFailed() = true}}, the live bucket still at 1 file instead of 2, and
the {{getBucketLayout()}} NPE from both sub-tasks on process and retry. A
passing test means the defect is present.
h3. Patch
[^MC-5-skip-key-events-of-deleted-bucket.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
A key event whose bucket is no longer in the Recon OM DB is skipped in both
sub-tasks, as the reprocess path already does. The bucket and its NSSummary
entry are unreachable after the delete, so nothing is lost. Covered by
{{TestNSSummaryTask.TestProcessParallelMixedBatch#testKeyEventOfDeletedBucketDoesNotFailBatch}}
(bucket not cached) and {{#testKeyEventOfDeletedCachedBucketDoesNotFailBatch}}
(cached bucket, bucketTable DELETE and keyTable DELETE in one batch), which
both fail without the change. With it, the NSSummary task suites pass (54
tests) and checkstyle is clean. With the patch the reproduction fails because
the batch succeeds and the live bucket counts both keys.
Related: HDDS-10855 covered the same root cause for the Legacy parent directory
lookup ({{setKeyParentID}}) and only changed its messages; that path still
fails a batch when a parent directory key is deleted in the same delta and is
not changed here.
Found by TLA+ model checking and code review of the Recon namespace summary and
task sync paths under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
