Siyao Meng created HDDS-16894:
---------------------------------
Summary: OneReplicaPipelineSafeModeRule still counts reports of
pipelines closed during safe mode after its refresh lowers the threshold
Key: HDDS-16894
URL: https://issues.apache.org/jira/browse/HDDS-16894
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch,
TestBugMC3OneReplicaRuleCountsClosedPipeline.java
h3. Mechanism
While the rule is not met, {{OneReplicaPipelineSafeModeRule.refresh(false)}}
calls {{initializeRule(true)}}, which rebuilds {{oldPipelineIDSet}} and
{{thresholdCount}} from the RATIS/THREE pipelines that are open now.
{{currentReportedPipelineCount}} and {{reportedPipelineIDSet}} are kept as they
are, so reports of pipelines that have closed since still count towards a
threshold that only covers open pipelines. With R reported pipelines of which C
have closed, the rule passes once R reaches ceil(0.9 x open), so it covers C
fewer open pipelines than configured. Once {{SCMSafeModeManager}} records the
rule as validated it is never checked again.
The class Javadoc says the rule ensures at least one reported datanode per open
pipeline, so that open containers have a replica to read when SCM leaves safe
mode. The container rules only count CLOSED and QUASI_CLOSED containers.
h3. Trigger
# SCM starts in safe mode with open RATIS/THREE pipelines, and the datanodes of
some of them report.
# One of the reported pipelines is closed while SCM is in safe mode: by
{{StaleNodeHandler}} or {{DeadNodeHandler}} after a registered datanode goes
silent, by a Ratis pipeline action from a datanode, or by {{ozone admin
pipeline close}}.
# The periodic refresh ({{hdds.scm.safemode.rule.refresh.interval}}, default
5s) runs. In releases before the periodic refresh thread, the refresh runs on
{{SCMStateMachine.applyTransaction}} in safe mode, so the transaction that
closes the pipeline triggers it (code reading, not run on a release).
h3. Impact
* Smallest case, run: two open pipelines p1 and p2 (threshold 2). Only p1's
three datanodes register and report, and p1 is reported healthy. p1 is closed.
The refresh sets the threshold to 1 with the count still 1, the rule validates
with 0 of 1 open pipelines reported, and in the same refresh cycle SCM leaves
safe mode, because every other rule already passes (3 datanodes registered, one
healthy pipeline, no closed containers).
* SCM leaves safe mode only if the other rules pass. If the unreported
datanodes hold closed containers with keys, the container rule still holds SCM
in safe mode unless those containers are under 1% of the total. The rule's own
bound already allows 10% of open pipelines to stay unreported. The bug loosens
that by one pipeline per reported pipeline closed during safe mode.
* After an early exit, {{getContainerWithPipeline}} no longer refuses open
containers without reported replicas, so reads of those containers go to
datanodes that have not reported until they come back, and block allocation can
pick the unreported open pipeline (code reading, not run). No data loss is
expected.
* The refresh path dates from HDDS-5263 (1.2.0). Related: HDDS-11844 proposes
changing what this rule requires before SCM leaves safe mode. This report and
patch keep the current requirement.
h3. Reproduction
[^TestBugMC3OneReplicaRuleCountsClosedPipeline.java]
({{hadoop-hdds/server-scm}}) runs the smallest case above with the real
{{PipelineManagerImpl}}, the real {{SCMSafeModeManager}} with all its rules and
its refresh thread (interval lowered to 100ms), real registration, pipeline
report and open pipeline events, and {{PipelineManagerImpl.closePipeline}}. On
the unmodified commit it fails every time with {{SCM left safe mode with 0 of 1
open pipelines reported}}, and the log shows {{OneReplicaPipelineSafeModeRule
rule is successfully validated}} followed by {{SCM exiting safe mode.}} With
the patch it passes. Not run on a real cluster.
h3. Patch
[^MC-3-drop-closed-pipeline-reports-on-one-replica-rule-refresh.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
6818707698c.
On refresh, {{initializeRule(true)}} keeps only the reported pipelines that are
still open ({{reportedPipelineIDSet.retainAll(oldPipelineIDSet)}}) and lowers
{{currentReportedPipelineCount}} by the number removed. Reports of pipelines
that are still open are kept. The count is lowered rather than set to the set
size because {{cleanup()}} empties the set after the rule has passed. From the
code, this also covers the report free validation path
({{updateReportedPipelineSet}}), which uses the same set and count. That path
is not tested. Nothing changes when no reported pipeline closes during safe
mode. The metric {{currentPipelinesWithAtleastOneReplicaReportedCount}} is a
counter and still includes the closed pipelines.
Covered by {{testRefreshDropsReportsOfPipelinesNoLongerOpen}} in the existing
{{TestOneReplicaPipelineSafeModeRule}}: three open pipelines, two report, one
of those closes, and after the refresh the rule expects threshold 2 and count 1
and is not met. A report from the third pipeline then satisfies it. Without the
change it fails ({{expected: <1> but was: <2>}}). With it, the
{{org.apache.hadoop.hdds.scm.safemode}} unit tests (75) and
{{TestSCMSafeModeWithPipelineRules}} on a mini cluster (2) pass, and checkstyle
is clean.
Found by TLA+ model checking and code review of the SCM safe mode rules under
HDDS-15926, on commit b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against
HDDS issues and apache/ozone pull requests for duplicates before filing. The
attached patch is a proposal for review. Generated with Specula (Claude Opus
5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
