[
https://issues.apache.org/jira/browse/HDDS-16922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy Huang reassigned HDDS-16922:
---------------------------------
Assignee: Andy Huang
> SCM terminates with exit status 0 when it shuts itself down on a fatal error,
> such as a failed Ratis snapshot install
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16922
> URL: https://issues.apache.org/jira/browse/HDDS-16922
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Assignee: Andy Huang
> Priority: Minor
> Attachments: CR-2-exit-with-failure-status-on-scm-shutdown.patch,
> TestBugCR2FatalInstallFailureExitStatus.java
>
>
> h3. Mechanism
> {{StorageContainerManager.shutDown(String)}} stops SCM and then calls
> {{ExitUtils.terminate(0, message, LOG)}}. Ratis {{ExitUtils}} logs a status 0
> termination at INFO and calls {{System.exit(0)}}. HDDS-6930 set the status to
> 0 because a normal stop then went through {{SCMStateMachine.close}} and this
> method. Since HDDS-10643, {{SCMStateMachine.close}} skips {{shutDown}} when
> SCM is already stopping, so a normal stop no longer reaches this method that
> way. All remaining callers are fatal error paths:
> * {{SCMStateMachine.close}} when Ratis closes the server on its own, for
> example after {{reinitialize}} fails during a snapshot install.
> * {{RootCARotationManager}} (9 sites) and {{RootCARotationHandlerImpl}} (3
> sites) on root CA rotation failures.
> * The uncaught exception handler of {{BackgroundPipelineCreator}}.
> A normal stop can still get here in one narrow case:
> {{StorageContainerManager.stop}} interrupts an in-flight root CA rotation
> task; its next Ratis request then fails, and the error handling calls
> {{shutDown}}. In production the stop runs in the JVM shutdown hook, so the
> process exit status is already decided there and only the log level of the
> termination changes (ERROR instead of INFO). This is from code reading, not
> run.
> Other fatal SCM paths already exit with status 1, for example an
> {{applyTransaction}} failure or the reload failure in
> {{SCMHAManagerImpl.installCheckpoint}}.
> h3. Trigger
> Any of the failures above. Reproduced with a failed snapshot install: a
> follower SCM that is behind the leader's purged log starts, and moving its
> old scm.db aside fails ({{HAUtils.replaceDBWithCheckpoint}}: "Failed to
> create a backup of the current DB"), here because its metadata directory is
> read only.
> h3. Impact
> The SCM process exits with status 0 after a fatal error, and the termination
> is logged at INFO ("Terminating with exit status 0: scm statemachine is
> closed by ratis, terminate SCM"). A supervisor that restarts only on failure
> (systemd {{Restart=on-failure}}, Kubernetes {{restartPolicy: OnFailure}})
> treats it as a clean stop and leaves the SCM down, and alerting on the exit
> status misses it. Supervisors that always restart are not affected. The
> status was recorded by Ratis {{ExitUtils}} with {{System.exit}} disabled in
> the mini cluster; no SCM process was run under a supervisor.
> h3. Reproduction
> PASS on an unmodified checkout.
> [^TestBugCR2FatalInstallFailureExitStatus.java]
> ({{hadoop-ozone/integration-test}}) uses the setup of
> {{TestSCMInstallSnapshotWithHA}}: 3 SCMs of which 2 start, log purge after 5
> entries. The leader advances past index 200, the inactive SCM's metadata
> directory is made read only, and that SCM is started. The install fails and
> Ratis {{ExitUtils}} records the first termination with status 0, logged at
> INFO. The mini cluster disables {{System.exit}}, so the status is the one
> that would have been passed to it. The test does not work as root. A passing
> test means the defect is present.
> h3. Patch
> [^CR-2-exit-with-failure-status-on-scm-shutdown.patch], against
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. It also applies to master at
> 52812559dad.
> {{StorageContainerManager.shutDown}} now terminates with status 1, so Ratis
> also logs the termination at ERROR. Since HDDS-10643, the normal stop that
> HDDS-6930 addressed no longer reaches this method through
> {{SCMStateMachine.close}}, and the new test checks that:
> {{testStateMachineClosedByRatisTerminatesWithFailureStatus}} in the existing
> {{TestStorageContainerManager}} stops one SCM normally and asserts that
> nothing terminated. It then has the state machine of a second SCM closed
> while Ratis is still running and asserts status 1. Without the change it
> fails with "expected: 1 but was: 0". With it, the test and the neighbouring
> {{TestStorageContainerManager}} tests pass, as do the SCM state machine, HA
> manager, root CA rotation and pipeline creator unit tests (49) and
> {{TestSCMInstallSnapshotWithHA}} and {{TestFailoverWithSCMHA}} on a mini
> cluster. Checkstyle is clean. With the patch the reproduction fails at its
> status assertion, with status 1 logged at ERROR.
> {{OzoneManager.shutDown}} (used when Ratis shuts down the OM state machine)
> has the same pattern; it is left for a separate follow up and not changed
> here.
> Found by code review of the SCM Ratis snapshot and checkpoint install paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> b0aa6475b7856d2c7f467dd3db12b9a1dd116f63. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
