Siyao Meng created HDDS-16917:
---------------------------------
Summary: SCM that crashes while it applies a root CA rotation
commit cannot start again, the sub CA directory switch is not completed on start
Key: HDDS-16917
URL: https://issues.apache.org/jira/browse/HDDS-16917
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-4-complete-interrupted-rotation-commit-on-scm-start.patch,
TestBugMC4CrashBetweenCommitMoves.java
h3. Mechanism
{{RootCARotationHandlerImpl.rotationCommit}} switches to the new sub CA
certificate in three separate steps: move {{scm/sub-ca}} to
{{scm/sub-ca-previous}}, move {{scm/sub-ca-next}} to {{scm/sub-ca}}, write the
new certificate ID to the VERSION file. SCM start reads only the VERSION file
and {{scm/sub-ca}} and has no step that completes or reverts a switch that
stopped in between.
* Stopped after the first move: {{scm/sub-ca}} does not exist.
{{DefaultCAServer.init}} for the sub CA throws {{IllegalStateException:
INTERMEDIARY_CA Should not be in Initialize State during startup.}} in the
{{StorageContainerManager}} constructor, on the primary SCM and on any other
SCM.
* Stopped after the second move: {{scm/sub-ca}} holds the new certificate, the
VERSION file names the old one. {{SCMCertificateClient.getCertificate}} returns
null. On the primary SCM the constructor fails with a {{NullPointerException}}
in {{persistPrimarySCMCerts}}. On any other SCM the constructor returns without
a certificate, and {{RootCARotationManager.shouldSkipRootCert}}, the first call
of the commit when it is applied again, throws a {{NullPointerException}} on
the missing trust chain. Inside {{SCMStateMachine.applyTransaction}} that
terminates the SCM (the replay itself is argued, not run).
The commit can never be applied again to finish the switch, because the SCM
fails before or while it replays the RAFT log.
The same three steps are in the release tags ozone-1.4.0, 1.4.1, 2.0.0, 2.1.0,
2.1.1, 2.1.2, 2.2.0 and 2.2.1, and SCM start does not look at
{{scm/sub-ca-previous}} in any of them (read with git grep, not run).
h3. Trigger
Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} (default false). An
SCM process dies (kill, OOM, power loss) between the first directory move and
the VERSION write of {{rotationCommit}}. The window is two renames and one
small file write, once per rotation and SCM. With the default
{{hdds.x509.max.duration}} of P1865D a rotation runs once in about five years.
The second state is also reached without a process death in that window. An SCM
that is restarted between its prepare ack and the commit terminates after both
moves and before the VERSION write (HDDS-16913). An {{IOException}} from the
VERSION write makes the handler call {{scm.shutDown}} on the thread that
applies the log entry, which leaves the SCM hung until it is killed
(HDDS-16911), and the next start then finds the second state (argued, not run).
h3. Impact
The SCM fails on every later start until an operator moves the directories or
edits the VERSION file by hand. No key or certificate is lost: the old pair is
in {{scm/sub-ca-previous}} and the new pair in {{scm/sub-ca-next}} or
{{scm/sub-ca}}. {{ozone repair scm cert recover}} does not handle these
directories.
If the SCM was a follower, the other SCMs finish the rotation and the ring runs
with one member less. If it was the leader, the other SCMs still apply the
commit, but the committed request, which only the leader sends and only right
after the commit, is not sent (read, not run). The rotation is then also left
unfinished on the other SCMs, see HDDS-16912.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugMC4CrashBetweenCommitMoves.java] ({{hadoop-ozone/integration-test}})
runs {{scm --init}} of a secure SCM with a MiniKdc, stages a new sub CA
certificate with the real certificate client and a new root CA (the leader that
signs the request is a mock backed by that CA), performs the directory moves of
{{rotationCommit}} up to the crash point and then constructs the
{{StorageContainerManager}} again. Both crash points fail as described above. A
passing test means the defect is present.
The test SCM is the primary. For an SCM that is not the primary the same states
were given to the real constructor after the primary SCM ID in the VERSION file
was changed (run, not part of the attached reproduction): after the first move
the constructor throws the same {{IllegalStateException}}, after the second
move it returns with {{getCertificate()}} null and {{shouldSkipRootCert}}
throws the {{NullPointerException}}.
h3. Patch
[^MC-4-complete-interrupted-rotation-commit-on-scm-start.patch], against
b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
5108acff480. The patch is made against that commit alone, other fixes proposed
for the same code may need a manual merge.
Before the SCM certificate client is created,
{{StorageContainerManager.initializeCertificateClient}} calls the new
{{RootCARotationManager.completeInterruptedCommit}}. It acts only when the
certificate named in the VERSION file is found in {{scm/sub-ca-previous}} and
is not the one in {{scm/sub-ca}}, which only an unfinished commit leaves
behind. It then moves {{scm/sub-ca-next}} to {{scm/sub-ca}} if that has not
happened, writes the ID of the certificate in {{scm/sub-ca}} to the VERSION
file and deletes {{scm/sub-ca-previous}}. The switch is completed and not
reverted because the commit entry is already committed in the RAFT log. When
the commit and committed entries are applied again they are skipped, since the
SCM already trusts the new root certificate. If {{scm/sub-ca}} is missing and
nothing is staged in {{scm/sub-ca-next}} the directories are left as they are
and the start fails as before (run). A certificate that is staged but was never
committed is not touched.
Covered by four cases of {{testScmStartCompletesInterruptedRotationCommit}} in
the existing {{TestSecureOzoneCluster}}: the two crash points, a certificate
that is staged and not committed, and a {{scm/sub-ca-previous}} that is a copy
of the directory in use. In the last two the SCM must start on its old
certificate and leave the directories alone. Without the change the two crash
point cases fail with the two exceptions above. With it the new test,
{{testSecureScmStartupFailure}} of the same class,
{{TestRootCARotationManager}} and {{TestScmInvokerCodeGenerator}} pass (18
tests) and checkstyle is clean. The other tests of {{TestSecureOzoneCluster}}
were not run. With the patch both cases of the reproduction fail, because the
SCM starts.
Found by TLA+ model checking and code review of the SCM root CA rotation paths
under HDDS-15926, on commit b9b33851b20fa11baf4b07982de9afb8ea44f85a. Checked
against HDDS issues and apache/ozone pull requests for duplicates before
filing. The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
