Siyao Meng created HDDS-16922:
---------------------------------
Summary: SCM terminates with exit status 0 when it shuts itself
down on a fatal error, such as a failed Ratis snapshot install
Key: HDDS-16922
URL: https://issues.apache.org/jira/browse/HDDS-16922
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-2-exit-with-failure-status-on-scm-shutdown.patch,
TestBugCR2FatalInstallFailureExitStatus.java
h3. Mechanism
{{StorageContainerManager.shutDown(String)}} stops SCM and then calls
{{ExitUtils.terminate(0, message, LOG)}}. Ratis {{ExitUtils}} logs a status 0
termination at INFO and calls {{System.exit(0)}}. HDDS-6930 set the status to 0
because a normal stop then went through {{SCMStateMachine.close}} and this
method. Since HDDS-10643, {{SCMStateMachine.close}} skips {{shutDown}} when SCM
is already stopping, so a normal stop no longer reaches this method that way.
All remaining callers are fatal error paths:
* {{SCMStateMachine.close}} when Ratis closes the server on its own, for
example after {{reinitialize}} fails during a snapshot install.
* {{RootCARotationManager}} (9 sites) and {{RootCARotationHandlerImpl}} (3
sites) on root CA rotation failures.
* The uncaught exception handler of {{BackgroundPipelineCreator}}.
A normal stop can still get here in one narrow case:
{{StorageContainerManager.stop}} interrupts an in-flight root CA rotation task;
its next Ratis request then fails, and the error handling calls {{shutDown}}.
In production the stop runs in the JVM shutdown hook, so the process exit
status is already decided there and only the log level of the termination
changes (ERROR instead of INFO). This is from code reading, not run.
Other fatal SCM paths already exit with status 1, for example an
{{applyTransaction}} failure or the reload failure in
{{SCMHAManagerImpl.installCheckpoint}}.
h3. Trigger
Any of the failures above. Reproduced with a failed snapshot install: a
follower SCM that is behind the leader's purged log starts, and moving its old
scm.db aside fails ({{HAUtils.replaceDBWithCheckpoint}}: "Failed to create a
backup of the current DB"), here because its metadata directory is read only.
h3. Impact
The SCM process exits with status 0 after a fatal error, and the termination is
logged at INFO ("Terminating with exit status 0: scm statemachine is closed by
ratis, terminate SCM"). A supervisor that restarts only on failure (systemd
{{Restart=on-failure}}, Kubernetes {{restartPolicy: OnFailure}}) treats it as a
clean stop and leaves the SCM down, and alerting on the exit status misses it.
Supervisors that always restart are not affected. The status was recorded by
Ratis {{ExitUtils}} with {{System.exit}} disabled in the mini cluster; no SCM
process was run under a supervisor.
h3. Reproduction
PASS on an unmodified checkout. [^TestBugCR2FatalInstallFailureExitStatus.java]
({{hadoop-ozone/integration-test}}) uses the setup of
{{TestSCMInstallSnapshotWithHA}}: 3 SCMs of which 2 start, log purge after 5
entries. The leader advances past index 200, the inactive SCM's metadata
directory is made read only, and that SCM is started. The install fails and
Ratis {{ExitUtils}} records the first termination with status 0, logged at
INFO. The mini cluster disables {{System.exit}}, so the status is the one that
would have been passed to it. The test does not work as root. A passing test
means the defect is present.
h3. Patch
[^CR-2-exit-with-failure-status-on-scm-shutdown.patch], against
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
52812559dad.
{{StorageContainerManager.shutDown}} now terminates with status 1, so Ratis
also logs the termination at ERROR. Since HDDS-10643, the normal stop that
HDDS-6930 addressed no longer reaches this method through
{{SCMStateMachine.close}}, and the new test checks that:
{{testStateMachineClosedByRatisTerminatesWithFailureStatus}} in the existing
{{TestStorageContainerManager}} stops one SCM normally and asserts that nothing
terminated. It then has the state machine of a second SCM closed while Ratis is
still running and asserts status 1. Without the change it fails with "expected:
1 but was: 0". With it, the test and the neighbouring
{{TestStorageContainerManager}} tests pass, as do the SCM state machine, HA
manager, root CA rotation and pipeline creator unit tests (49) and
{{TestSCMInstallSnapshotWithHA}} and {{TestFailoverWithSCMHA}} on a mini
cluster. Checkstyle is clean. With the patch the reproduction fails at its
status assertion, with status 1 logged at ERROR.
{{OzoneManager.shutDown}} (used when Ratis shuts down the OM state machine) has
the same pattern; it is left for a separate follow up and not changed here.
Found by code review of the SCM Ratis snapshot and checkpoint install paths, as
part of the TLA+ verification effort under HDDS-15926, on commit
b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
