Siyao Meng created HDDS-17050:
---------------------------------
Summary: Recon task cursor advances past a batch that failed both
attempts, so a restart soon after the failure can lose that batch
Key: HDDS-17050
URL: https://issues.apache.org/jira/browse/HDDS-17050
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-skip-batches-behind-failed-batch-until-reinit.patch,
TestBugMC7FailedBatchLostAcrossRestart.java
h3. Mechanism
When a delta batch fails both attempts in
{{ReconTaskControllerImpl.processOMUpdateBatch}}, the controller only sets the
in memory {{tasksFailed}} flag; the next sync iteration turns it into a
reinitialization. Batches that are already in the event buffer behind the
failed one are still processed, and for each task that succeeds
{{processTasks}} saves {{lastUpdatedSeqNumber}} to the task status table. The
cursor of the failed task therefore reaches the cursor of {{OmDeltaRequest}}.
If Recon restarts before the reinitialization has finished and before the sync
has fetched any new OM update, the flag and the queued reinitialization are
gone, and the startup check in {{OzoneManagerServiceProviderImpl.start}}, which
rebuilds only tasks whose cursor is below that of {{OmDeltaRequest}}, finds
nothing to do. The failed batch is never applied.
The same happens when a manual rebuild drains a buffered batch and the
rebuild's reprocess step then fails, because that failure also only sets
{{tasksFailed}} (code reading, not run). A rebuild that fails in its DB swap
sets no flag at all, because {{reInitializeTasks}} still reports success;
HDDS-17047 makes it report the failure.
h3. Trigger
# The sync buffers two batches before the event thread gets to the first, as
when it fetches several deltas in one run.
# A task fails the first batch on both attempts and succeeds on the second
batch. The failure has to be tied to the first batch; a Recon RocksDB I/O error
closes the DB, so the second batch would fail too. One such failure is
HDDS-17048: {{NSSummaryTask}} fails with a NullPointerException, on both
attempts, on a key event whose bucket was deleted later in the same delta,
while a later batch about other buckets succeeds (code reading).
# Recon restarts after the event thread has processed the buffered batches and
before the sync fetches any new OM update. A non empty fetch after the failure
moves the {{OmDeltaRequest}} cursor past the task cursors, because the batch is
refused or drained, and the startup check then rebuilds. On a cluster with
ongoing writes this leaves about one sync interval (5s by default). Without new
writes, it lasts until the queued reinitialization has completed.
h3. Impact
The changes of the failed batch are missing from that task's derived data for
good: for NSSummary the namespace summary and du APIs report too few files and
bytes. Later syncs and restarts do not notice it; a manual rebuild repairs it.
Without a restart the reinitialization repairs it as designed.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC7FailedBatchLostAcrossRestart.java] ({{hadoop-ozone/recon}}) uses a
real OM RocksDB that serves its write ahead log through a mocked
{{getDBUpdates}}, and the real {{OzoneManagerServiceProviderImpl}},
{{ReconTaskControllerImpl}}, {{NSSummaryTask}} and task status table.
{{NSSummaryTask.process}} returns a failed result for one batch on both
attempts, and its first attempt is held until the next batch is buffered. The
restart stops the provider and controller and starts new ones on the same
databases. Output before the restart: {{tasksFailed}} true, NSSummaryTask
cursor 5, OmDeltaRequest cursor 5, OM has 3 files under the bucket and
NSSummary 2. After the restart and a sync NSSummary still reports 2; a rebuild
gives 3. A passing test means the defect is present.
h3. Patch
[^MC-7-skip-batches-behind-failed-batch-until-reinit.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
{{processOMUpdateBatch}} does not run the tasks while {{tasksFailed}} is set.
In that state {{consumeOMEvents}} already refuses new batches, and queueing the
reinitialization drains the buffer, clears the flag and takes the checkpoint
the rebuild uses, which contains every batch applied to the Recon OM DB, so the
skipped batches would be discarded by it anyway. Not processing them keeps the
failed task's cursor behind, and the existing startup check rebuilds after a
restart. Without a restart nothing changes, except that until the
reinitialization is queued (normally the next sync iteration) the tasks that
did not fail also stop applying buffered batches, which the reinitialization
rebuilds.
Covered by {{testBatchesBehindFailedBatchDoNotAdvanceTaskCursor}} in the
existing {{TestReconTaskControllerImpl}}, which fails without the change
({{expected: 0L but was: 200L}}). With the change the Recon tasks suites and
{{TestOzoneManagerServiceProviderImpl}} (155 tests) pass and checkstyle is
clean. With the patch the reproduction fails at its undercount assertion: the
NSSummaryTask cursor stays at 3, the restart rebuilds the tasks and NSSummary
reports 3.
A related case, a task whose {{process}} throws instead of returning a failed
result and so never sets {{tasksFailed}}, is HDDS-17040.
Found by TLA+ model checking and code review of the Recon namespace summary and
task sync paths under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-5146 (a failure between the Recon OM DB update and the task
hand off loses the batch) and HDDS-16463 (sync the Recon DB before the cursor
advances). The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
