Siyao Meng created HDDS-16906:
---------------------------------
Summary: Pipeline with a DEAD datanode stays OPEN if SCM
leadership changes while the node goes STALE and DEAD, and its EC container is
not repaired
Key: HDDS-16906
URL: https://issues.apache.org/jira/browse/HDDS-16906
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-7-scrub-open-pipelines-with-dead-nodes.patch,
TestBugMC7DeadEcPipelineMemberAfterFailovers.java
h3. Mechanism
Every SCM, leader or follower, runs its own node health check and fires
{{STALE_NODE}} and {{DEAD_NODE}} once per transition. {{StaleNodeHandler}} and
{{DeadNodeHandler}} close the node's pipelines through
{{PipelineManager.closePipeline}}, which is a Ratis request: on a follower it
fails with NotLeaderException. {{StaleNodeHandler}} logs "Could not finalize
pipeline" at INFO without the exception, {{DeadNodeHandler}} logs the
exception, and both drop it. No SCM fires the events again for a node that is
already DEAD, and nothing re-runs the handlers when an SCM becomes leader. The
only periodic check, {{PipelineManagerImpl.scrubPipelines}} on the leader,
closes an OPEN pipeline only if a member is not registered (HDDS-7341); a DEAD
datanode stays registered.
So if the SCM that becomes leader saw the STALE transition while it was a
follower, and the same happens for the DEAD transition, no SCM closes the
pipeline. For RATIS/THREE pipelines the remaining datanodes can still trigger
the close through pipeline actions (argued, not run). A RATIS/ONE pipeline on
the dead datanode and EC pipelines have no such path, because no other member
is left or because EC datanodes are not aware of pipelines.
h3. Trigger
# An OPEN EC pipeline, three SCMs, and a datanode of the pipeline stops.
# The datanode goes STALE. A follower sees it first, its handler fails, and
that follower becomes leader before the old leader's close has been applied.
# The datanode goes DEAD. Again the SCM that becomes leader saw it first as a
follower.
Each SCM checks node health on its own timer
({{ozone.scm.heartbeat.thread.interval}}, default 3s), so each leader change
has to fall in a window of a few seconds around the transition, at both
transitions (5 minutes apart with the default stale and dead intervals). A
close that fails on the leader for another reason is lost the same way (argued,
not run).
h3. Impact
* The EC pipeline stays OPEN with a DEAD member.
{{SCMBlockProtocolServer.allocateBlock}} keeps returning it to requests that do
not exclude it. The client then fails over: it adds the pipeline to its exclude
list and asks for another one (argued from the client code, not run).
* Its OPEN container keeps 4 of 5 replicas. {{ReplicationManager}} does not
reconstruct OPEN containers, and the container cannot fill, so it stays under
replicated until the datanode returns, an admin closes the pipeline or
container, or an SCM that was restarted after the node died becomes leader
(then the node is unregistered there and the existing scrubber rule closes the
pipeline).
* In the same run, two RATIS/THREE pipelines and one RATIS/ONE pipeline with
the dead datanode were also still OPEN on the new leader after a scrubber pass.
Whether the datanodes close the RATIS/THREE ones later was not observed.
h3. Reproduction
PASS, unmodified source. [^TestBugMC7DeadEcPipelineMemberAfterFailovers.java]
({{hadoop-ozone/integration-test}}) starts 3 SCMs, 1 OM and 5 datanodes with
the stale and dead intervals shortened to 6s and 40s, and writes one EC 3-2 key
with the client. It widens the two windows with the existing
{{SCMNodeManager.pauseHealthCheck}} test hook on the current leader and moves
leadership with the admin {{transferLeadership}} RPC: followers B and C see
STALE, leadership moves A to B, followers A and C see DEAD, leadership moves B
to C. On C it then runs one scrubber pass and one ReplicationManager pass.
Output on the pin: the EC pipeline is OPEN with a DEAD member, its container is
OPEN with 4 replicas, {{allocateBlock}} for EC 3-2 returns that pipeline, and
the log shows the stale node handler's "Could not finalize pipeline" lines and
the dead node handler failing with NotLeaderException on the followers. As a
control, a second member stopped while C is leader gets the pipeline closed. A
passing test means the defect is present. With the patch it fails at the first
check, because the scrubber pass closes the pipeline and the container moves to
CLOSING.
h3. Patch
[^MC-7-scrub-open-pipelines-with-dead-nodes.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
b9b33851b20.
{{scrubPipelines}} also closes an OPEN pipeline that has a DEAD member, next to
the existing rule for unregistered members. The scrubber runs only on the
leader, so the close no longer depends on which SCM saw the transition, and it
also covers a close that failed on a stable leader. {{closePipeline}} closes
the pipeline's containers too, so ReplicationManager can repair them once they
are closed. The scrubber runs only on a leader that is ready and out of safe
mode. In normal operation the leader's stale or dead node handler has already
closed every pipeline of a DEAD node, of any type, so the rule finds nothing.
If such a close was lost, the scrubber does it at most one scrub interval
(default 5 minutes) after the node is DEAD. It does not act on STALE members.
Covered by {{testScrubOpenWithDeadNodes}} in the existing
{{TestPipelineManagerImpl}} (a STALE member keeps the pipeline open, a DEAD
member closes it), which fails without the change. With it the SCM pipeline
unit suites, {{TestSCMSafeModeManager}}, {{TestSCMNodeManager}} and
{{TestDeadNodeHandler}} (359 tests) and {{TestDecommissionAndMaintenance}} (9
tests) pass, checkstyle is clean.
Related: HDDS-16887. If {{DeadNodeHandler}} deletes the pipeline during a
scrubber pass, the new close hits the {{PipelineNotFoundException}} that
HDDS-16887 handles (today it ends the pass, as it already can for the existing
rules). HDDS-16887 also wraps the loop body in {{scrubPipelines}} in a {{try}}
block, so the patches conflict textually and whichever lands second needs a
trivial rebase.
Found by TLA+ model checking and code review of the SCM pipeline lifecycle
paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
Checked against HDDS issues and apache/ozone pull requests for duplicates
before filing. The attached patch is a proposal for review. Generated with
Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
