[
https://issues.apache.org/jira/browse/HDDS-17224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17224:
------------------------------
Attachment:
MC-2-wait-for-full-container-report-before-admin-completion.patch
> Decommission or maintenance completes on an empty container set after SCM
> restart or datanode re-registration, before the full container report
> -----------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17224
> URL: https://issues.apache.org/jira/browse/HDDS-17224
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-2-wait-for-full-container-report-before-admin-completion.patch,
> TestBugMC2EmptyContainerSetPassesGate.java
>
>
> h3. Mechanism
> {{DatanodeAdminMonitorImpl.checkContainersReplicatedOnNode}} checks only the
> containers in {{NodeManager.getContainers}} for the node and passes when that
> set is empty. The set is filled only by container reports, and it starts
> empty whenever the node gets a new {{DatanodeEntry}}:
> * {{NodeStateMap.addNode}}, when a datanode registers with an SCM that has
> restarted;
> * {{NodeStateMap.updateNode}}, when a datanode registers again with a new
> address, ports or software version (the container set loss described in
> HDDS-16632).
> {{SCMDatanodeProtocolServer.register}} fires the full container report as an
> event only after {{SCMNodeManager.register}} has returned, and the report is
> processed on the container report executor. The monitor runs on its own
> executor and does not check safe mode or whether a report has been processed.
> After an SCM restart, the admin workflow is resumed for a node that registers
> with a persisted DECOMMISSIONING or ENTERING_MAINTENANCE state, by the
> NEW_NODE handler ({{NodeDecommissionManager.continueAdminForNode}}) or, if
> the node registered before this SCM became leader, by
> {{NodeDecommissionManager.onBecomeLeader}}. The other gate conditions do not
> hold the node: its pipelines were closed when the workflow started, and its
> persisted state matches. If a monitor run lands before the report is
> processed, the node completes the workflow. The report that arrives later
> does not reopen it.
> h3. Trigger
> Re-registration:
> # A datanode is being decommissioned and the monitor is waiting for an under
> replicated container on it.
> # The datanode restarts and registers again with a new software version
> (rolling upgrade) or a changed port set (seen on a plain datanode restart in
> a mini cluster in HDDS-17190). Its container set in SCM is now empty.
> # A monitor run happens before the full container report from the
> registration has been processed.
> # The node becomes DECOMMISSIONED.
> SCM restart:
> # A datanode is DECOMMISSIONING or ENTERING_MAINTENANCE and holds an under
> replicated container. SCM restarts.
> # The datanode registers with its persisted state, and the leader starts
> tracking it again.
> # A monitor run happens before its full container report has been processed
> (after an SCM restart many datanodes register and their reports queue up).
> # The node becomes DECOMMISSIONED or IN_MAINTENANCE.
> h3. Impact
> The node is reported DECOMMISSIONED (safe to remove) or IN_MAINTENANCE while
> containers on it are still under replicated, and it is no longer tracked.
> This can hold for every container on the node at once. Once the report is
> processed, Replication Manager sees the containers as under replicated and
> copies them from the node while it is still up, so redundancy is lost only if
> the operator removes or stops the node first, and data only with a further
> failure. The window is from registration until the report is processed; how
> long it is on a real cluster was not measured.
> h3. Reproduction
> PASS on unmodified source, deterministic.
> [^TestBugMC2EmptyContainerSetPassesGate.java] ({{hadoop-hdds/server-scm}})
> uses the real {{SCMNodeManager}} for registration, heartbeats and operational
> state, the real {{ContainerReportHandler}} for the full container report
> (with a stub {{ContainerManager}} that knows one CLOSED RATIS/THREE
> container) and the real {{DatanodeAdminMonitorImpl}}. Replication Manager is
> mocked as in {{TestDatanodeAdminMonitor}}: the container has the replica on
> the node and one other copy. The only thing the test controls is whether the
> monitor runs before or after the report is processed.
> * Re-registration: the node is DECOMMISSIONING and held by the container,
> registers again with version 2.0.0 instead of 1.0.0 (container set empty),
> the monitor runs, then the report is processed. The node is DECOMMISSIONED,
> and stays so with the container back in its set.
> * SCM restart, persisted DECOMMISSIONING and ENTERING_MAINTENANCE: with the
> report processed first the node stays in its admin state; with the monitor
> first it becomes DECOMMISSIONED or IN_MAINTENANCE.
> A passing test means the defect is present.
> h3. Patch
> [^MC-2-wait-for-full-container-report-before-admin-completion.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 20d95ed37fe.
> {{DatanodeInfo}} gets a flag that {{ContainerReportHandler}} sets once a full
> container report has been applied to the node's container set, and
> {{checkContainersReplicatedOnNode}} does not pass while the node's current
> {{DatanodeInfo}} lacks it. The flag needs no reset: SCM creates a new
> {{DatanodeInfo}} in exactly the two cases where the container set starts
> empty. Incremental reports do not set it. The change is on the gate side, so
> it is independent of how HDDS-16632 is fixed; carrying the container set over
> in {{NodeStateMap.updateNode}} alone would not cover the SCM restart case.
> This patch is meant to cover the admin monitor symptom that HDDS-16632
> describes, so a fix for HDDS-16632 does not need to change the monitor. The
> flag is read after the container set, so a registration between the two reads
> counts as a node still waiting for its report. Behaviour change: after an SCM
> restart or a re-registration, a tracked node waits for its first full
> container report to be processed before it can complete. Normally that is the
> report sent with the registration; if the handler fails on that one, the node
> waits for the next full report ({{hdds.container.report.interval}}, 60
> minutes by default) or until the datanode registers again, which sends a new
> report.
> Covered by {{testAdminWaitsForContainerReportAfterRegistration}}
> (decommission and maintenance) in the existing {{TestDatanodeAdminMonitor}},
> and {{testFullReportMarksContainerReportProcessed}} in the existing
> {{TestIncrementalContainerReportHandler}} (an incremental report does not set
> the flag, a full report does, after re-registering with a new version the
> node starts without it). Without the gate change the first fails with
> "expected: <DECOMMISSIONING> but was: <DECOMMISSIONED>" and "expected:
> <ENTERING_MAINTENANCE> but was: <IN_MAINTENANCE>". With it
> {{TestDatanodeAdminMonitor}}, {{TestIncrementalContainerReportHandler}},
> {{TestContainerReportHandler}}, {{TestSCMNodeManager}},
> {{TestNodeDecommissionManager}}, {{TestNodeDecommissionMetrics}} and
> {{TestDeadNodeHandler}} pass, the mini cluster suite
> {{TestDecommissionAndMaintenance}} (9 tests, including its SCM restart cases)
> passes, checkstyle is clean, and the three reproduction tests fail because
> the node waits for the report and is then held by the under replicated
> container.
> Found by TLA+ model checking and code review of the SCM datanode decommission
> and maintenance admin paths under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-16632 (container set lost on re-registration, mentions this
> effect without a reproduction), HDDS-16925 (ContainerManager replicas lost on
> SCM snapshot install). The attached patch is a proposal for review. Generated
> with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
