Siyao Meng created HDDS-17042:
---------------------------------
Summary: Recon manual task rebuild requested while a delta sync is
being applied makes the tasks apply that delta twice
Key: HDDS-17042
URL: https://issues.apache.org/jira/browse/HDDS-17042
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-6-wait-for-delta-hand-off-before-manual-rebuild.patch
h3. Mechanism
{{OzoneManagerServiceProviderImpl.syncDataFromOM}} commits each delta fetched
from OM to the Recon OM DB (in {{innerGetAndApplyDeltaUpdatesFromOM}}) and only
afterwards writes the task status rows and hands the events to the tasks with
{{ReconTaskController.consumeOMEvents}}. {{triggerTaskRebuild}}, behind {{POST
/api/v1/triggerdbsync/om/reinit}}, calls {{queueReInitializationEvent}} without
any coordination with that sequence. If it runs during or after the commit and
before the hand off, it drains the event buffer (which does not hold the new
batch yet), takes a checkpoint of the Recon OM DB that already contains the
batch, or part of it, since one delta response is committed as several write
batches, and queues the reinitialization. The sync thread then buffers the
batch behind the reinitialization event, so the tasks first rebuild from a
checkpoint that includes all or part of the batch and then process the whole
batch again. The rebuild sets every task cursor to the checkpoint's sequence
number and nothing skips a batch at or below it, so the second application is
never noticed.
h3. Trigger
# OM writes a key; the scheduled delta sync of Recon fetches it and commits it
to the Recon OM DB.
# Before that sync hands the events to the tasks, an admin calls {{POST
/api/v1/triggerdbsync/om/reinit}}. The call returns 202.
# The rebuild runs, then the same events are applied on top of it.
The window is the time from the start of the commit of one delta response to
its hand off, so it is short per sync, but under sustained OM writes the sync
applies deltas back to back. Only the REST rebuild can hit it: the NSSummary
rebuild of the upgrade actions runs before the OM sync is started.
h3. Impact
Every event of the batch is applied twice by every task. For NSSummary, file
counts and sizes of the affected directories and buckets are wrong (too high
for a replayed put, too low for a replayed delete), and the namespace summary
and du APIs report them. The file size, container key and table insight counts
are affected by the same mechanism (code reading, only NSSummary was run). The
task cursors match OM, so later syncs and a restart do not correct it; only
another rebuild outside the window does.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC6RebuildDuringSyncDoubleCounts.java] ({{hadoop-ozone/recon}}) uses a
real OM RocksDB that serves its write ahead log through a mocked
{{getDBUpdates}}, and the real {{OzoneManagerServiceProviderImpl}},
{{ReconTaskControllerImpl}} and {{NSSummaryTask}} with the staged rebuild and
swap. The sync thread is paused in {{updateOMMetadataManager}}, which the sync
calls between the commit and the hand off, and the rebuild is requested from a
second thread. Output: OM has 2 files under the bucket and NSSummary reports 3;
after one more ordinary sync OM has 3 and NSSummary 4. A passing test means the
defect is present.
h3. Patch
[^MC-6-wait-for-delta-hand-off-before-manual-rebuild.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
A lock in {{OzoneManagerServiceProviderImpl}} is taken by the sync thread after
the OM RPC has returned and before it commits the delta, and released at the
end of that sync iteration, after the hand off. {{triggerTaskRebuild}} takes it
around {{queueReInitializationEvent}}. The rebuild therefore checkpoints either
before the delta is committed (the batch is then applied after the rebuild,
once) or after the batch is in the event buffer (it is drained and contained in
the checkpoint). The lock is never held during an OM RPC, so a rebuild while OM
is unreachable, the bootstrapping case the endpoint exists for, is not delayed.
Refusing the rebuild while a sync is running would be simpler but would block
that case, because a sync against an unreachable OM spends a long time in RPC
retries.
Covered by {{testTriggerTaskRebuildWaitsForDeltaHandOff}} in the existing
{{TestOzoneManagerServiceProviderImpl}}; it fails without the change because
the rebuild is queued inside the window. With the change the Recon spi, tasks
and sync endpoint suites (200 tests) and {{TestReconWithOzoneManager}} on a
mini cluster, including the manual rebuild with OM stopped, pass, and
checkstyle is clean. With the patch the reproduction fails at its in window
assertion ({{rebuildQueuedInWindow}}), before the count assertions: the rebuild
waits for the hand off and NSSummary reports 2 and then 3, matching OM. Totals
that are already doubled are not repaired.
Found by TLA+ model checking and code review of the Recon namespace summary and
task sync paths under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-15863 (added the manual rebuild) and HDDS-5146 (a failure
between the Recon OM DB update and the task hand off loses the batch). The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
