Siyao Meng created HDDS-17071:
---------------------------------
Summary: Recon PipelineSyncTask stops after one failed cycle and
ContainerHealthTask retries a failed sweep without waiting
Key: HDDS-17071
URL: https://issues.apache.org/jira/browse/HDDS-17071
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch,
TestBugMC6ReconTaskLoopAfterFailedCycle.java
h3. Mechanism
Both are periodic Recon tasks built on {{ReconScmTask}}, whose
{{initializeAndRunTask}} passes every exception up to the task's {{run}} loop.
* {{PipelineSyncTask.run}} has its {{catch (Throwable)}} outside the {{while
(canRun())}} loop. When one cycle fails, the thread logs "Exception in Pipeline
sync Thread." and ends. The task's running flag stays true, so {{isRunning()}}
keeps reporting the task as running and {{start()}} would not start it again;
nothing else does either. A cycle fails when {{scmClient.getPipelines()}} or
{{getNodes()}} throws: Recon's SCM client retries are bounded
({{ozone.recon.scmclient.max.retry.timeout}} 6s,
{{ozone.recon.scmclient.failover.max.retry}} 3), so an SCM outage longer than a
few seconds reaches the task as an {{IOException}}. It also fails when a write
to Recon's own SCM DB fails; HDDS-10017 shows this exception from a real
cluster, where {{ReconPipelineManager.initializePipelines}} failed on such a
write.
* {{ContainerHealthTask.run}} catches per cycle, but its interval sleep is
inside the {{try}} after the sweep. When the sweep throws, for example because
the Recon SQL database rejects the write of {{UNHEALTHY_CONTAINERS}}, the next
full sweep starts at once instead of after the interval (at least 60s).
The {{PipelineSyncTask}} loop has this shape in every release since ozone-1.0.0
(read from the tagged source). The health task hot loop came with HDDS-13891
(ozone-2.2.0 and 2.2.1); in earlier releases that thread instead ended on its
first failure, like the pipeline sync.
h3. Trigger
* Pipeline sync: any cycle that fails (default every 5 minutes), such as during
an SCM restart, upgrade or failover, or on an error from Recon's local SCM DB.
* Health task: any error from the sweep, for example an outage of an external
Recon SQL database ({{ozone.recon.sql.db.jdbc.url}}).
h3. Impact
* After the pipeline sync thread has ended, Recon no longer refreshes the
operational state of DEAD datanodes from SCM, and no longer closes or removes
pipelines that SCM has closed or removed, until Recon restarts. The sync still
runs when some datanode goes STALE or DEAD (those handlers call it directly),
so a busy cluster may hide the problem while a quiet one does not. Live
datanodes are not affected because they report their state in heartbeats.
* A datanode that is DEAD while in maintenance keeps its replicas in Recon
({{DeadNodeHandler}} skips the removal for nodes in maintenance). When SCM ends
the maintenance, the periodic sync is what copies IN_SERVICE to Recon, which
fires DEAD_NODE again and removes the replicas. With the thread ended, Recon
keeps showing the node IN_MAINTENANCE and keeps its replicas. In the attached
run with real {{ReconNodeManager}}, {{PipelineSyncTask}} and
{{ReconDeadNodeHandler}}, after 25 sync intervals SCM had IN_SERVICE, Recon had
IN_MAINTENANCE, the task had made no further SCM call while {{isRunning()}}
returned true, and the replica was never removed. That the kept replica then
hides the container from Recon's under replicated and missing reports is argued
from the code, not run.
* A failing health sweep runs back to back, each failure logging a stack trace,
for as long as the failure lasts. The attached run counted about 20000 failing
sweeps (mocked, failing at once) in about one second. On a real cluster each
failed sweep still walks every container before the database write fails, so
the sweeps run back to back at the speed of a full sweep.
h3. Reproduction
[^TestBugMC6ReconTaskLoopAfterFailedCycle.java] goes in {{hadoop-ozone/recon}}
(target path and Maven command in the class comment) and passes on unmodified
source when the bugs are present. The first test uses a real
{{ReconNodeManager}} with short stale and dead intervals, a real
{{PipelineSyncTask}} with a 200ms interval and a real {{ReconDeadNodeHandler}};
only the SCM RPC client is mocked (its first {{getPipelines()}} throws
{{IOException}}) and the container manager is a mock so the replica removal can
be observed. The second runs the real {{ContainerHealthTask}} thread over a
mocked {{ReconReplicationManager}} whose {{processAll}} throws a jOOQ
{{DataAccessException}}.
h3. Patch
[^MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch], against
916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
d77edaa6c15.
{{PipelineSyncTask.run}} now catches {{Exception}} per cycle, logs the same
error, records the failed run in the task status and sleeps the interval before
the next cycle. {{ContainerHealthTask.run}} moves its existing catch directly
after the sweep so the interval sleep follows a failed sweep too. In both, an
interrupt still ends the loop: the per cycle catch restores the interrupt flag
of an {{InterruptedException}}, so the sleep that follows throws and the loop
exits as before. An {{Error}} still ends the thread, as it does in
{{ContainerHealthTask}} today. This is the per cycle pattern the SCM container
sync and OM sync schedulers in Recon already use.
Covered by the new {{TestPipelineSyncTask.keepsRunningAfterFailedSync}} (first
{{getPipelines()}} fails, a later run must succeed) and by
{{TestContainerHealthTask.testFailedRunWaitsForNextInterval}} (a failing sweep
must run once in the first second). Without the change the first sees only the
failed run and the second counted about 20000 sweeps. With it the Recon {{scm}}
and {{fsck}} unit tests pass, checkstyle is clean, and both attached tests no
longer show the bugs.
{{ContainerSizeCountTask}} has the same catch outside its loop; it is tracked
in HDDS-17065.
Found by TLA+ model checking and code review of the datanode container report
and Recon container tracking paths under HDDS-15926, on commit
916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-5331 (resolved, added the stale and dead node triggers for
this task), HDDS-10017 (Cannot Reproduce, a field log of this exception after a
failed write to Recon's SCM DB), HDDS-5814 (resolved, interrupt handling in
these loops), HDDS-13891 (resolved, introduced the current
{{ContainerHealthTask}} loop). The attached patch is a proposal for review.
Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
