Siyao Meng created HDDS-17046:
---------------------------------
Summary: An Error during the Recon NSSummary rebuild leaves the
rebuild state RUNNING, so every later rebuild is skipped until restart
Key: HDDS-17046
URL: https://issues.apache.org/jira/browse/HDDS-17046
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-6-mark-rebuild-failed-on-error.patch,
TestBugCR6RebuildStateAfterError.java
h3. Mechanism
{{NSSummaryTask.reprocess}} sets the static {{REBUILD_STATE}} to RUNNING and
calls {{executeReprocess}} inside a {{catch (Exception e)}} that sets FAILED.
{{executeReprocess}} sets FAILED or IDLE only on the failures it handles itself
and on success. A {{java.lang.Error}} thrown on the reprocess thread passes
through all of these and leaves RUNNING behind. From then on every
{{reprocess}} call returns failure with "NSSummary tree rebuild is already in
progress, skipping duplicate request." Nothing outside tests resets the state.
A realistic source is {{Executors.newFixedThreadPool(3).invokeAll}} in
{{executeReprocess}}, which throws {{OutOfMemoryError: unable to create native
thread}} when the process cannot create threads. Errors inside a sub-task do
not cause this; they arrive wrapped in an {{ExecutionException}}, which sets
FAILED.
h3. Trigger
A Recon rebuild ({{ReconTaskControllerImpl.reInitializeTasks}}) runs while the
JVM throws an Error on the rebuild thread, for example a transient native
thread or heap exhaustion. The {{CompletableFuture}} absorbs the Error and
Recon keeps running; the default launch scripts do not exit on OutOfMemoryError.
h3. Impact
* Every later NSSummary rebuild is refused until Recon restarts.
* Argued from the code, not run: {{reInitializeTasks}} then returns false on
every attempt, the controller keeps its task failure flag set,
{{consumeOMEvents}} drops new event batches and the OM sync loop queues another
rebuild on each run. So all Recon OM derived data stops updating until restart.
Each attempt reprocesses all other tasks into a staging DB that is then
discarded, and after repeated failures the sync loop falls back to downloading
a full OM DB snapshot from OM, which queues another rebuild that is refused
again, so the snapshot downloads repeat as well.
h3. Reproduction
PASS on unmodified source, deterministic.
[^TestBugCR6RebuildStateAfterError.java] ({{hadoop-ozone/recon}}) runs
{{reInitializeTasks}} on a real controller with the unmodified
{{NSSummaryTask}} over a real Recon OM DB. The Error is injected once, as
{{OutOfMemoryError("unable to create native thread")}} from the first call of
the rebuild ({{ReconNamespaceSummaryManager.clearNSSummaryTable}}, a mock).
After it the state is RUNNING; the next rebuild returns false without running
and the state stays RUNNING. A passing test means the defect is present. Real
native thread exhaustion was not attempted.
h3. Patch
[^CR-6-mark-rebuild-failed-on-error.patch], against
0665932dd35da4d05087e02748af0a64eb846ad5. It also applies to master at
2264d4d48ac.
{{reprocess}} also catches {{Error}}, sets FAILED and rethrows it, so the
controller sees the same failure as before and the next rebuild can run.
Covered by {{TestNSSummaryUnifiedControl#testRebuildAfterError}}, which fails
without the change ({{expected: FAILED but was: RUNNING}}). With it,
{{TestNSSummaryUnifiedControl}}, the NSSummary task suites and
{{TestReconTaskControllerImpl}} pass (83 tests, 1 skipped as before), as do the
two upgrade action tests that read the state, and checkstyle is clean. With the
patch the reproduction fails because the state is FAILED after the Error and
the next rebuild runs.
Found by code review of the Recon namespace summary and task sync paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
0665932dd35da4d05087e02748af0a64eb846ad5. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
