Chu Cheng Li created HDDS-17364:
-----------------------------------
Summary: SCM stays in safe mode when containers go to DELETING
while it replays its Ratis log
Key: HDDS-17364
URL: https://issues.apache.org/jira/browse/HDDS-17364
Project: Apache Ozone
Issue Type: Bug
Components: SCM
Reporter: Chu Cheng Li
*Problem*
{{RatisContainerSafeModeRule}} and {{ECContainerSafeModeRule}} build their
container list in the SCM constructor. They take the CLOSED and QUASI_CLOSED
containers with keys from the SCM DB. SCM then starts Ratis and applies the log
entries after the last DB flush. An entry can move an empty container to
DELETING. Datanodes delete the replicas of a DELETING container. Thus, no
datanode reports it at registration.
The rule refresh from HDDS-15498 removes DELETED containers from the list. It
does not remove DELETING containers. Only the ReplicationManager of the leader
moves a container from DELETING to DELETED. The ReplicationManager does not run
in safe mode. Thus, when more than 1% of the containers in the list are
DELETING ({{hdds.scm.safemode.threshold.pct}} is 0.99 by default), SCM does not
leave safe mode.
*Effect*
SCM stays in safe mode until an admin forces it out. Block allocation and
replication stop.
In the 3-SCM simulation (HDDS-16627, seed 2064), an SCM restarted with a DB
flush from before 4 containers went to DELETING. It became leader before they
went to DELETED. It stayed in safe mode until the end of the run, and 4
decommissions did not finish.
>From the code, one SCM can also get this. It must restart, for example after a
>crash, after containers go to DELETING and before they go to DELETED, with its
>last DB flush from before the DELETING entries.
*Fix*
Remove DELETING containers in the refresh too. A trial change of 2 lines in
{{AbstractContainerSafeModeRule#reinitializeRule}} passes the new test, the
existing safe mode tests (57) and simulation seed 2064. With it and a trial fix
for HDDS-17363, the simulation passes 100 of 100 seeds with 1 SCM and 100 of
100 with 3 SCMs.
*Tests*
{{AbstractContainerSafeModeRuleTest#testRefreshDropsDeletingContainers}} runs
for both rules. It fails on master with "expected: <1.0> but was: <0.5>".
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
