[
https://issues.apache.org/jira/browse/HDDS-16903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16903:
------------------------------
Attachment: TestBugMC4NonOpenPipelineCountsAsHealthy.java
> HealthyPipelineSafeModeRule counts DORMANT and CLOSED pipelines, so SCM can
> leave safe mode with no healthy OPEN RATIS/THREE pipeline
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16903
> URL: https://issues.apache.org/jira/browse/HDDS-16903
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Minor
> Attachments:
> MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch,
> TestBugMC4NonOpenPipelineCountsAsHealthy.java
>
>
> h3. Mechanism
> While SCM is in safe mode, {{PipelineReportHandler.processPipelineReport}}
> fires {{OPEN_PIPELINE}} for every reported pipeline that is healthy (all
> members reported, leader known), whatever its state in SCM.
> {{HealthyPipelineSafeModeRule.process}}, the only subscriber, checks the
> replication config, whether the pipeline was already counted, the member
> count and the member health, but not the state. A DORMANT or CLOSED
> RATIS/THREE pipeline whose datanodes still host the Ratis group is therefore
> counted, although the threshold in {{initializeRule}} is computed from OPEN
> pipelines only. The count never goes down, and once every rule has passed SCM
> does not return to safe mode.
> The other checks in the same area count OPEN pipelines only:
> {{OneReplicaPipelineSafeModeRule.process}} checks {{isOpen()}}, and so does
> the report free path
> {{validateHealthyPipelineSafeModeRuleUsingPipelineManager}} (HDDS-13590).
> HDDS-4399 ("Safe mode rule for pipelines should only consider open
> pipelines") moved the {{fireEvent}} out of the ALLOCATED branch so that
> pipelines reloaded as OPEN are counted, and the state check was lost with
> that move.
> h3. Trigger
> Datanodes keep a pipeline's Ratis group, and keep reporting it, until SCM
> sends a close command. For a pipeline that was deactivated, or closed by a
> pipeline action, an admin close or {{StaleNodeHandler}}, SCM sends it only
> when the pipeline scrubber deletes the CLOSED pipeline, and the scrubber runs
> only outside safe mode. Two ways to get there:
> * DORMANT: an admin runs {{ozone admin pipeline deactivate}}. Only the SCM
> state changes, and the state is persisted, so the pipeline counts at every
> later SCM restart for as long as its datanodes are up.
> * CLOSED: during safe mode a member asks SCM to close the pipeline (pipeline
> action, for example after a Ratis failure), an admin runs {{ozone admin
> pipeline close}}, or a member goes stale for a while (for example a long GC
> pause) and {{StaleNodeHandler}} closes its pipelines before it recovers. The
> members keep reporting the pipeline for the rest of safe mode. A pipeline
> closed shortly before an SCM restart is loaded as CLOSED and behaves the same
> (argued, not run).
> SCM leaves safe mode on such a pipeline once the other rules pass, for
> example after a restart where an OPEN pipeline has only one live member: that
> one member satisfies {{OneReplicaPipelineSafeModeRule}}, but the pipeline
> cannot become healthy.
> h3. Impact
> SCM leaves safe mode although no OPEN RATIS/THREE pipeline is healthy. That
> is the situation this rule exists to prevent (HDDS-1178). Block allocation is
> then allowed, and new RATIS/THREE writes can only proceed once a new pipeline
> is created and opened. Where the datanodes have room for a new pipeline, this
> mostly makes the exit earlier. Where they have none (for example, the slots
> are held by the DORMANT pipeline), writes keep failing until an operator acts
> (argued from the code, not run). No data is lost, and the container rules
> still gate the exit. The state check is missing since 1.1.0 (HDDS-4399).
> h3. Reproduction
> [^TestBugMC4NonOpenPipelineCountsAsHealthy.java] ({{hadoop-hdds/server-scm}})
> uses the real {{PipelineManagerImpl}}, {{PipelineReportHandler}},
> {{PipelineActionHandler}} and {{SCMSafeModeManager}} with all its rules,
> driven only by datanode registration, pipeline report and pipeline action
> events. Pipeline P1 is OPEN with one live member. Pipeline P2 is on three
> other live datanodes and is deactivated before the start (first test), or
> closed by a member's close action during safe mode (second test). Before P2's
> members report it, only {{HealthyPipelineSafeModeRule}} is unmet. After their
> reports, SCM leaves safe mode with P1 as the only OPEN RATIS/THREE pipeline
> and P1 not healthy. Both tests pass on unmodified source, which means the
> defect is present. With the patch, both fail because SCM stays in safe mode.
> The reproduction does not model datanodes opening pipelines that the
> background pipeline creator allocates.
> h3. Patch
> [^MC-4-count-only-open-pipelines-in-healthy-pipeline-rule.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> a5aeb764d80.
> {{HealthyPipelineSafeModeRule.process}} reads the pipeline's current state
> from {{PipelineManager}} and skips it unless it is OPEN, as
> {{OneReplicaPipelineSafeModeRule}} does. The check is in the rule rather than
> in {{PipelineReportHandler}} for two reasons. The {{Pipeline}} object the
> handler fires is the copy it read before calling {{openPipeline}}, so it
> still says ALLOCATED for a pipeline that the same report has just opened. And
> the event is processed asynchronously. {{PipelineReportHandler}} is unchanged.
> One behaviour change: if the only fully reported RATIS/THREE pipelines are
> DORMANT or CLOSED and no OPEN one can become healthy, SCM now stays in safe
> mode until one does or an operator runs {{ozone admin safemode exit}},
> instead of leaving it without a writable pipeline. With the default
> {{hdds.scm.safemode.pipeline.creation=true}} a CLOSED pipeline only delays
> the exit while three healthy datanodes remain: CLOSED pipelines do not count
> against the datanode pipeline limit, so the background creator builds a new
> one. The case that changes is DORMANT pipelines taking every free pipeline
> slot, for example an admin who deactivates all pipelines and then restarts
> SCM. SCM then stays in safe mode, so {{ReplicationManager}} does not run
> either, until the admin activates a pipeline or runs {{ozone admin safemode
> exit}}.
> Covered by {{testPipelineNotOpenIsIgnored}} in the existing
> {{TestHealthyPipelineSafeModeRule}}: with one OPEN, one DORMANT and one
> CLOSED pipeline, all fully reported, the rule must not count the DORMANT or
> the CLOSED pipeline, and must pass on the OPEN one. The events carry stale
> copies, as the handler's may: an OPEN copy of the DORMANT and of the CLOSED
> pipeline, and the ALLOCATED copy of the OPEN pipeline, so a check on the
> event's own copy fails the test as well. Without the change it fails
> ({{expected: 0 but was: 1}}). With it, the
> {{org.apache.hadoop.hdds.scm.safemode}} unit tests together with
> {{TestPipelineManagerImpl}} and {{TestPipelineActionHandler}} (108 tests)
> pass, as do {{TestSCMSafeModeWithPipelineRules}} and
> {{TestSafeModeCheckSubcommandHA}} on a mini cluster (8 tests), and checkstyle
> is clean.
> Found by TLA+ model checking and code review of the SCM safe mode rules under
> HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. This issue
> also covers the same defect found by code review of the SCM pipeline
> lifecycle under HDDS-16538, which approached it from
> {{PipelineReportHandler}} and the DORMANT state. Checked against HDDS issues
> and apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-4399 (introduced the state independent event), HDDS-13590
> (OPEN filter for the report free path only), HDDS-11904 (datanode health
> check in the same method) and HDDS-11844 (open, proposes not waiting for all
> pipelines). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
