[
https://issues.apache.org/jira/browse/HDDS-16905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16905:
------------------------------
Attachment: TestBugMC6StaleNode2PipelineAfterSnapshotInstall.java
> SCM keeps deleted pipelines in its datanode to pipeline map after a Ratis
> snapshot install, so decommission and maintenance stall while it is leader
> ----------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16905
> URL: https://issues.apache.org/jira/browse/HDDS-16905
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-6-remove-stale-node-pipelines-on-scm-reinitialize.patch,
> TestBugMC6StaleNode2PipelineAfterSnapshotInstall.java
>
>
> h3. Mechanism
> When a follower SCM installs a snapshot from the leader,
> {{SCMHAManagerImpl.startServices}} calls
> {{PipelineStateManagerImpl.reinitialize}}. It replaces the
> {{PipelineStateMap}} with an empty one and {{initialize}} adds every pipeline
> of the new pipeline table to the state map and to the NodeManager
> ({{nodeManager.addPipeline}}). The pipelines of the old map are never removed
> from the NodeManager. The datanode to pipeline map ({{Node2PipelineMap}}) is
> only updated by {{PipelineStateManagerImpl}}, and its only removal path,
> {{PipelineStateManagerImpl.removePipeline}}, needs the pipeline to be in the
> state map. So a pipeline that was closed and deleted while the follower was
> behind stays listed for its datanodes until that SCM process restarts.
> {{DatanodeAdminMonitorImpl.checkPipelinesClosedOnNode}}, the first gate for
> decommission and for entering maintenance, waits until
> {{nodeManager.getPipelines(dn)}} is empty, and decommission has no expiry.
> {{StartDatanodeAdminHandler}} cannot close the stale pipeline either, because
> the pipeline manager does not know it.
> h3. Trigger
> # {{ozone.scm.ha.ratis.log.purge.enabled=true}} (default false). Without log
> purge the leader keeps its whole log and sends a lagging follower entries
> instead of a snapshot (read from code).
> # A follower SCM knows pipeline P. It is stopped (or falls behind) while the
> leader closes and deletes P and purges its Raft log past the follower's last
> index.
> # The follower catches up by installing a snapshot
> ({{SCMStateMachine.reinitialize}}). Its NodeManager still lists P for P's
> datanodes.
> # That SCM becomes leader and an operator decommissions a datanode of P.
> Only the restart variant of step 2 was run. A live follower that is sent a
> snapshot takes the same path (argued from code).
> h3. Impact
> * The decommission of every datanode that was in such a pipeline stays in
> DECOMMISSIONING without an error while that SCM is leader. The monitor logs
> "Waiting for pipelines to close ... There are 1 pipelines" on every run.
> Restarting that SCM clears the stale entries (read from code, not run).
> Moving leadership to another SCM should also let the decommission continue,
> since the other SCMs have a correct map (argued, not run).
> * A datanode entering maintenance waits on the stale entry the same way,
> until its maintenance end time passes, or forever if no end time was given
> (argued, not run).
> * {{NodeManager.getPipelinesCount}} and the pipeline count in datanode usage
> info include the stale entries (argued, not run). Pipeline placement is not
> affected, because it ignores pipeline IDs the state manager does not know.
> h3. Reproduction
> PASS, unmodified source.
> [^TestBugMC6StaleNode2PipelineAfterSnapshotInstall.java]
> ({{hadoop-ozone/integration-test}}) starts 3 SCMs, 1 OM and 3 datanodes with
> log purge enabled and the small purge gap and snapshot threshold that
> {{TestSCMInstallSnapshotWithHA}} uses. It stops a follower, closes an open
> RATIS/THREE pipeline with the admin close pipeline RPC and lets the leader's
> scrubber delete it, advances the Raft log by 200 entries with in process
> {{ContainerManager.allocateContainer}} calls on the leader, restarts the
> follower, which installs a snapshot, transfers leadership to it and
> decommissions one datanode of the deleted pipeline. Output on the pin: after
> the snapshot install the restarted SCM does not know the pipeline but lists
> it for the datanode, and after 8 monitor runs the datanode is still
> DECOMMISSIONING with exactly that pipeline left. A passing test means the
> defect is present. With the patch it fails at the first check after the
> snapshot install, because the datanode then lists only its live pipelines.
> h3. Patch
> [^MC-6-remove-stale-node-pipelines-on-scm-reinitialize.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> b9b33851b20.
> {{PipelineStateManagerImpl.reinitialize}} removes the pipelines of the old
> state map from the NodeManager before it rebuilds the map; {{initialize}}
> adds the pipelines of the new table back as before. The NodeManager then
> lists exactly the pipelines of the new state map. Recon's SCM DB snapshot
> sync ({{ReconStorageContainerManagerFacade.initializeNewRdbStore}}) calls the
> same {{reinitialize}} and gets the same fix (read from code, not run).
> Covered by {{testReinitializeRemovesPipelinesMissingFromNewStore}} in the
> existing {{TestPipelineStateManagerImpl}} (two pipelines on one datanode,
> reinitialize with a table that keeps one), which fails without the change.
> With it the pipeline and node unit suites touched by this path (132 tests)
> and {{TestSCMInstallSnapshotWithHA}} and {{TestSCMInstallSnapshot}} pass,
> checkstyle is clean. Entries left on an SCM that already installed a snapshot
> are only cleared by a restart.
> Found by TLA+ model checking and code review of the SCM pipeline lifecycle
> paths under HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
