[
https://issues.apache.org/jira/browse/HDDS-17071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17071:
------------------------------
Attachment: TestBugMC6ReconTaskLoopAfterFailedCycle.java
> Recon PipelineSyncTask stops after one failed cycle and ContainerHealthTask
> retries a failed sweep without waiting
> ------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17071
> URL: https://issues.apache.org/jira/browse/HDDS-17071
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch,
> TestBugMC6ReconTaskLoopAfterFailedCycle.java
>
>
> h3. Mechanism
> Both are periodic Recon tasks built on {{ReconScmTask}}, whose
> {{initializeAndRunTask}} passes every exception up to the task's {{run}} loop.
> * {{PipelineSyncTask.run}} has its {{catch (Throwable)}} outside the {{while
> (canRun())}} loop. When one cycle fails, the thread logs "Exception in
> Pipeline sync Thread." and ends. The task's running flag stays true, so
> {{isRunning()}} keeps reporting the task as running and {{start()}} would not
> start it again; nothing else does either. A cycle fails when
> {{scmClient.getPipelines()}} or {{getNodes()}} throws: Recon's SCM client
> retries are bounded ({{ozone.recon.scmclient.max.retry.timeout}} 6s,
> {{ozone.recon.scmclient.failover.max.retry}} 3), so an SCM outage longer than
> a few seconds reaches the task as an {{IOException}}. It also fails when a
> write to Recon's own SCM DB fails; HDDS-10017 shows this exception from a
> real cluster, where {{ReconPipelineManager.initializePipelines}} failed on
> such a write.
> * {{ContainerHealthTask.run}} catches per cycle, but its interval sleep is
> inside the {{try}} after the sweep. When the sweep throws, for example
> because the Recon SQL database rejects the write of {{UNHEALTHY_CONTAINERS}},
> the next full sweep starts at once instead of after the interval (at least
> 60s).
> The {{PipelineSyncTask}} loop has this shape in every release since
> ozone-1.0.0 (read from the tagged source). The health task hot loop came with
> HDDS-13891 (ozone-2.2.0 and 2.2.1); in earlier releases that thread instead
> ended on its first failure, like the pipeline sync.
> h3. Trigger
> * Pipeline sync: any cycle that fails (default every 5 minutes), such as
> during an SCM restart, upgrade or failover, or on an error from Recon's local
> SCM DB.
> * Health task: any error from the sweep, for example an outage of an external
> Recon SQL database ({{ozone.recon.sql.db.jdbc.url}}).
> h3. Impact
> * After the pipeline sync thread has ended, Recon no longer refreshes the
> operational state of DEAD datanodes from SCM, and no longer closes or removes
> pipelines that SCM has closed or removed, until Recon restarts. The sync
> still runs when some datanode goes STALE or DEAD (those handlers call it
> directly), so a busy cluster may hide the problem while a quiet one does not.
> Live datanodes are not affected because they report their state in heartbeats.
> * A datanode that is DEAD while in maintenance keeps its replicas in Recon
> ({{DeadNodeHandler}} skips the removal for nodes in maintenance). When SCM
> ends the maintenance, the periodic sync is what copies IN_SERVICE to Recon,
> which fires DEAD_NODE again and removes the replicas. With the thread ended,
> Recon keeps showing the node IN_MAINTENANCE and keeps its replicas. In the
> attached run with real {{ReconNodeManager}}, {{PipelineSyncTask}} and
> {{ReconDeadNodeHandler}}, after 25 sync intervals SCM had IN_SERVICE, Recon
> had IN_MAINTENANCE, the task had made no further SCM call while
> {{isRunning()}} returned true, and the replica was never removed. That the
> kept replica then hides the container from Recon's under replicated and
> missing reports is argued from the code, not run.
> * A failing health sweep runs back to back, each failure logging a stack
> trace, for as long as the failure lasts. The attached run counted about 20000
> failing sweeps (mocked, failing at once) in about one second. On a real
> cluster each failed sweep still walks every container before the database
> write fails, so the sweeps run back to back at the speed of a full sweep.
> h3. Reproduction
> [^TestBugMC6ReconTaskLoopAfterFailedCycle.java] goes in
> {{hadoop-ozone/recon}} (target path and Maven command in the class comment)
> and passes on unmodified source when the bugs are present. The first test
> uses a real {{ReconNodeManager}} with short stale and dead intervals, a real
> {{PipelineSyncTask}} with a 200ms interval and a real
> {{ReconDeadNodeHandler}}; only the SCM RPC client is mocked (its first
> {{getPipelines()}} throws {{IOException}}) and the container manager is a
> mock so the replica removal can be observed. The second runs the real
> {{ContainerHealthTask}} thread over a mocked {{ReconReplicationManager}}
> whose {{processAll}} throws a jOOQ {{DataAccessException}}.
> h3. Patch
> [^MC-6-keep-recon-scm-tasks-running-after-failed-cycle.patch], against
> 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
> d77edaa6c15.
> {{PipelineSyncTask.run}} now catches {{Exception}} per cycle, logs the same
> error, records the failed run in the task status and sleeps the interval
> before the next cycle. {{ContainerHealthTask.run}} moves its existing catch
> directly after the sweep so the interval sleep follows a failed sweep too. In
> both, an interrupt still ends the loop: the per cycle catch restores the
> interrupt flag of an {{InterruptedException}}, so the sleep that follows
> throws and the loop exits as before. An {{Error}} still ends the thread, as
> it does in {{ContainerHealthTask}} today. This is the per cycle pattern the
> SCM container sync and OM sync schedulers in Recon already use.
> Covered by the new {{TestPipelineSyncTask.keepsRunningAfterFailedSync}}
> (first {{getPipelines()}} fails, a later run must succeed) and by
> {{TestContainerHealthTask.testFailedRunWaitsForNextInterval}} (a failing
> sweep must run once in the first second). Without the change the first sees
> only the failed run and the second counted about 20000 sweeps. With it the
> Recon {{scm}} and {{fsck}} unit tests pass, checkstyle is clean, and both
> attached tests no longer show the bugs.
> {{ContainerSizeCountTask}} has the same catch outside its loop; it is tracked
> in HDDS-17065.
> Found by TLA+ model checking and code review of the datanode container report
> and Recon container tracking paths under HDDS-15926, on commit
> 916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-5331 (resolved, added the stale and dead node triggers for
> this task), HDDS-10017 (Cannot Reproduce, a field log of this exception after
> a failed write to Recon's SCM DB), HDDS-5814 (resolved, interrupt handling in
> these loops), HDDS-13891 (resolved, introduced the current
> {{ContainerHealthTask}} loop). The attached patch is a proposal for review.
> Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
