[
https://issues.apache.org/jira/browse/HDDS-17046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17046:
------------------------------
Attachment: CR-6-mark-rebuild-failed-on-error.patch
> An Error during the Recon NSSummary rebuild leaves the rebuild state RUNNING,
> so every later rebuild is skipped until restart
> -----------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17046
> URL: https://issues.apache.org/jira/browse/HDDS-17046
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: CR-6-mark-rebuild-failed-on-error.patch,
> TestBugCR6RebuildStateAfterError.java
>
>
> h3. Mechanism
> {{NSSummaryTask.reprocess}} sets the static {{REBUILD_STATE}} to RUNNING and
> calls {{executeReprocess}} inside a {{catch (Exception e)}} that sets FAILED.
> {{executeReprocess}} sets FAILED or IDLE only on the failures it handles
> itself and on success. A {{java.lang.Error}} thrown on the reprocess thread
> passes through all of these and leaves RUNNING behind. From then on every
> {{reprocess}} call returns failure with "NSSummary tree rebuild is already in
> progress, skipping duplicate request." Nothing outside tests resets the state.
> A realistic source is {{Executors.newFixedThreadPool(3).invokeAll}} in
> {{executeReprocess}}, which throws {{OutOfMemoryError: unable to create
> native thread}} when the process cannot create threads. Errors inside a
> sub-task do not cause this; they arrive wrapped in an {{ExecutionException}},
> which sets FAILED.
> h3. Trigger
> A Recon rebuild ({{ReconTaskControllerImpl.reInitializeTasks}}) runs while
> the JVM throws an Error on the rebuild thread, for example a transient native
> thread or heap exhaustion. The {{CompletableFuture}} absorbs the Error and
> Recon keeps running; the default launch scripts do not exit on
> OutOfMemoryError.
> h3. Impact
> * Every later NSSummary rebuild is refused until Recon restarts.
> * Argued from the code, not run: {{reInitializeTasks}} then returns false on
> every attempt, the controller keeps its task failure flag set,
> {{consumeOMEvents}} drops new event batches and the OM sync loop queues
> another rebuild on each run. So all Recon OM derived data stops updating
> until restart. Each attempt reprocesses all other tasks into a staging DB
> that is then discarded, and after repeated failures the sync loop falls back
> to downloading a full OM DB snapshot from OM, which queues another rebuild
> that is refused again, so the snapshot downloads repeat as well.
> h3. Reproduction
> PASS on unmodified source, deterministic.
> [^TestBugCR6RebuildStateAfterError.java] ({{hadoop-ozone/recon}}) runs
> {{reInitializeTasks}} on a real controller with the unmodified
> {{NSSummaryTask}} over a real Recon OM DB. The Error is injected once, as
> {{OutOfMemoryError("unable to create native thread")}} from the first call of
> the rebuild ({{ReconNamespaceSummaryManager.clearNSSummaryTable}}, a mock).
> After it the state is RUNNING; the next rebuild returns false without running
> and the state stays RUNNING. A passing test means the defect is present. Real
> native thread exhaustion was not attempted.
> h3. Patch
> [^CR-6-mark-rebuild-failed-on-error.patch], against
> 0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
> 2264d4d48ac.
> {{reprocess}} also catches {{Error}}, sets FAILED and rethrows it, so the
> controller sees the same failure as before and the next rebuild can run.
> Covered by {{TestNSSummaryUnifiedControl#testRebuildAfterError}}, which fails
> without the change ({{expected: FAILED but was: RUNNING}}). With it,
> {{TestNSSummaryUnifiedControl}}, the NSSummary task suites and
> {{TestReconTaskControllerImpl}} pass (83 tests, 1 skipped as before), as do
> the two upgrade action tests that read the state, and checkstyle is clean.
> With the patch the reproduction fails because the state is FAILED after the
> Error and the next rebuild runs.
> Found by code review of the Recon namespace summary and task sync paths, as
> part of the TLA+ verification effort under HDDS-15926, on commit
> 0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
