Siyao Meng created HDDS-17066:
---------------------------------
Summary: Recon removes the replicas of a datanode stopped for
maintenance when it has not yet learned IN_MAINTENANCE from SCM
Key: HDDS-17066
URL: https://issues.apache.org/jira/browse/HDDS-17066
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-7-recon-take-dead-node-op-state-from-scm-before-replica-removal.patch,
TestBugMC7ReconDropsMaintenanceReplica.java
h3. Mechanism
{{ReconDeadNodeHandler.onMessage}} first calls {{DeadNodeHandler.onMessage}},
which keeps the replicas of a dead node only if the node is IN_MAINTENANCE, and
reads that state from Recon's own node manager. Only after that does it fetch
the node list from SCM and copy the node's operational state with
{{ReconNodeManager.updateNodeOperationalStateFromScm}}. HDDS-5277 added that
lookup to fix the displayed state; it was placed after the replica decision.
Recon learns the operational state of a live datanode only from the datanode's
heartbeats ({{ReconNodeManager.updateDatanodeOpState}}); {{PipelineSyncTask}}
copies it from SCM only for nodes that are already DEAD. The datanode gets
IN_MAINTENANCE in a heartbeat reply from SCM and reports it to Recon in its
next heartbeat. The datanode sends heartbeats to Recon on its SCM cycle
({{hdds.heartbeat.interval}}, 30s), so Recon can lag SCM by up to about 60s.
h3. Trigger
The documented maintenance workflow: {{ozone admin datanode maintenance}} for a
node, wait until SCM shows it IN_MAINTENANCE, stop the node. If the node is
stopped within about a minute, Recon still has ENTERING_MAINTENANCE (or
IN_SERVICE) when the node goes DEAD, and removes its replicas. A script that
stops the node as soon as SCM shows the state hits this. By code reading (not
run), the same happens if the node is stopped while SCM still shows
ENTERING_MAINTENANCE and SCM completes the transition before the node is DEAD,
since the admin workflow only aborts for a DEAD node.
h3. Impact
For the whole maintenance, Recon reports the node's containers UNDER_REPLICATED
while SCM, whose {{DeadNodeHandler}} keeps the replicas of an IN_MAINTENANCE
node, reports them healthy. In the attached run with 5 datanodes, after the
stopped node was DEAD, SCM listed 3 replicas including the stopped node and its
{{ReplicationManager}} report had no UNDER_REPLICATED container, while Recon
listed 2 replicas and its unhealthy containers endpoint returned one
UNDER_REPLICATED container. Recon showed ENTERING_MAINTENANCE for the node at
the moment it was stopped. The replicas come back only when the node returns
and reports. Monitoring only; SCM and the data are not affected.
h3. Reproduction
[^TestBugMC7ReconDropsMaintenanceReplica.java] goes in
{{hadoop-ozone/integration-test-recon}} (target path and Maven command in the
class comment) and passes on unmodified source when the bug is present. It
starts a mini cluster with 5 datanodes and Recon, writes a key, puts one
replica holder in maintenance through the SCM admin API
({{ScmClient.startMaintenanceNodes}}), waits until SCM shows IN_MAINTENANCE,
stops the datanode at once and waits until it is DEAD. It uses a 1s heartbeat
and short stale and dead intervals to keep the run short, and no hooks; Recon
still had ENTERING_MAINTENANCE when the node was stopped.
h3. Patch
[^MC-7-recon-take-dead-node-op-state-from-scm-before-replica-removal.patch],
against 916aa1d9949cbb10f2750a40c8704340893131a2. It also applies to master at
14e0611fdd5.
{{ReconDeadNodeHandler.onMessage}} now fetches the state from SCM first, then
calls {{DeadNodeHandler.onMessage}}, then runs the container health and
pipeline sync tasks. Recon then keeps the replicas of a node that SCM has
IN_MAINTENANCE, like SCM does. If the SCM call fails, the error is logged as
before, Recon decides with its own state, which is the current behaviour, and
the container health and pipeline sync tasks are skipped as before.
Covered by the new parameterized
{{testDeadNodeHandlerUsesOperationalStateFromScm}} in {{TestReconNodeManager}}:
SCM IN_MAINTENANCE keeps the replica, SCM IN_SERVICE and a failing SCM call
remove it, and the pipeline sync is skipped when the SCM call fails. Without
the change the first case fails with a replica removal that should not happen.
With it the unit tests in the Recon {{scm}} and {{fsck}} packages and
{{TestReconTasks}} pass, checkstyle is clean, and the attached reproduction
shows 3 replicas and no UNDER_REPLICATED container in Recon.
Found by TLA+ model checking and code review of the datanode container report
and Recon container tracking paths under HDDS-15926, on commit
916aa1d9949cbb10f2750a40c8704340893131a2. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. Related but not
duplicates: HDDS-5277 (resolved, added the SCM state lookup to this handler for
the displayed state), HDDS-2607 (resolved, SCM keeps the replicas of a dead
node in maintenance), HDDS-9645 (resolved, Recon counting of out of service
replicas). The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
