[
https://issues.apache.org/jira/browse/HDDS-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17042:
------------------------------
Attachment: MC-6-wait-for-delta-hand-off-before-manual-rebuild.patch
> Recon manual task rebuild requested while a delta sync is being applied makes
> the tasks apply that delta twice
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17042
> URL: https://issues.apache.org/jira/browse/HDDS-17042
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-6-wait-for-delta-hand-off-before-manual-rebuild.patch
>
>
> h3. Mechanism
> {{OzoneManagerServiceProviderImpl.syncDataFromOM}} commits each delta fetched
> from OM to the Recon OM DB (in {{innerGetAndApplyDeltaUpdatesFromOM}}) and
> only afterwards writes the task status rows and hands the events to the tasks
> with {{ReconTaskController.consumeOMEvents}}. {{triggerTaskRebuild}}, behind
> {{POST /api/v1/triggerdbsync/om/reinit}}, calls
> {{queueReInitializationEvent}} without any coordination with that sequence.
> If it runs during or after the commit and before the hand off, it drains the
> event buffer (which does not hold the new batch yet), takes a checkpoint of
> the Recon OM DB that already contains the batch, or part of it, since one
> delta response is committed as several write batches, and queues the
> reinitialization. The sync thread then buffers the batch behind the
> reinitialization event, so the tasks first rebuild from a checkpoint that
> includes all or part of the batch and then process the whole batch again. The
> rebuild sets every task cursor to the checkpoint's sequence number and
> nothing skips a batch at or below it, so the second application is never
> noticed.
> h3. Trigger
> # OM writes a key; the scheduled delta sync of Recon fetches it and commits
> it to the Recon OM DB.
> # Before that sync hands the events to the tasks, an admin calls {{POST
> /api/v1/triggerdbsync/om/reinit}}. The call returns 202.
> # The rebuild runs, then the same events are applied on top of it.
> The window is the time from the start of the commit of one delta response to
> its hand off, so it is short per sync, but under sustained OM writes the sync
> applies deltas back to back. Only the REST rebuild can hit it: the NSSummary
> rebuild of the upgrade actions runs before the OM sync is started.
> h3. Impact
> Every event of the batch is applied twice by every task. For NSSummary, file
> counts and sizes of the affected directories and buckets are wrong (too high
> for a replayed put, too low for a replayed delete), and the namespace summary
> and du APIs report them. The file size, container key and table insight
> counts are affected by the same mechanism (code reading, only NSSummary was
> run). The task cursors match OM, so later syncs and a restart do not correct
> it; only another rebuild outside the window does.
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC6RebuildDuringSyncDoubleCounts.java] ({{hadoop-ozone/recon}}) uses
> a real OM RocksDB that serves its write ahead log through a mocked
> {{getDBUpdates}}, and the real {{OzoneManagerServiceProviderImpl}},
> {{ReconTaskControllerImpl}} and {{NSSummaryTask}} with the staged rebuild and
> swap. The sync thread is paused in {{updateOMMetadataManager}}, which the
> sync calls between the commit and the hand off, and the rebuild is requested
> from a second thread. Output: OM has 2 files under the bucket and NSSummary
> reports 3; after one more ordinary sync OM has 3 and NSSummary 4. A passing
> test means the defect is present.
> h3. Patch
> [^MC-6-wait-for-delta-hand-off-before-manual-rebuild.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> A lock in {{OzoneManagerServiceProviderImpl}} is taken by the sync thread
> after the OM RPC has returned and before it commits the delta, and released
> at the end of that sync iteration, after the hand off. {{triggerTaskRebuild}}
> takes it around {{queueReInitializationEvent}}. The rebuild therefore
> checkpoints either before the delta is committed (the batch is then applied
> after the rebuild, once) or after the batch is in the event buffer (it is
> drained and contained in the checkpoint). The lock is never held during an OM
> RPC, so a rebuild while OM is unreachable, the bootstrapping case the
> endpoint exists for, is not delayed. Refusing the rebuild while a sync is
> running would be simpler but would block that case, because a sync against an
> unreachable OM spends a long time in RPC retries.
> Covered by {{testTriggerTaskRebuildWaitsForDeltaHandOff}} in the existing
> {{TestOzoneManagerServiceProviderImpl}}; it fails without the change because
> the rebuild is queued inside the window. With the change the Recon spi, tasks
> and sync endpoint suites (200 tests) and {{TestReconWithOzoneManager}} on a
> mini cluster, including the manual rebuild with OM stopped, pass, and
> checkstyle is clean. With the patch the reproduction fails at its in window
> assertion ({{rebuildQueuedInWindow}}), before the count assertions: the
> rebuild waits for the hand off and NSSummary reports 2 and then 3, matching
> OM. Totals that are already doubled are not repaired.
> Found by TLA+ model checking and code review of the Recon namespace summary
> and task sync paths under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-15863 (added the manual rebuild) and HDDS-5146 (a failure
> between the Recon OM DB update and the task hand off loses the batch). The
> attached patch is a proposal for review. Generated with Specula (Claude Opus
> 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
