Siyao Meng created HDDS-16915:
---------------------------------
Summary: SCM follower that installs a checkpoint taken after a
root CA rotation commit stays on the old root certificate
Key: HDDS-16915
URL: https://issues.apache.org/jira/browse/HDDS-16915
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-6-complete-prepared-rotation-after-checkpoint-install.patch,
TestBugMC6CheckpointSkipsRotation.java
h3. Mechanism
{{RootCARotationHandlerImpl.rotationCommit}} and {{rotationCommitted}} are
replicated requests, but what they change is local to each SCM: they move
{{scm/sub-ca-next}} to {{scm/sub-ca}}, write the new certificate ID to the
VERSION file and reload the certificate client. None of this is in the SCM DB.
When a follower catches up by installing a checkpoint,
{{SCMHAManagerImpl.installCheckpoint}} replaces the DB and reloads the DB
backed managers. The applied index moves past the two requests and the follower
never applies them. It keeps its sub CA certificate under the old root, and the
certificate it prepared for the new root stays in {{scm/sub-ca-next}}.
h3. Trigger
# Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} and
{{ozone.scm.ha.ratis.log.purge.enabled=true}} (both false by default).
# A root CA rotation starts. Every SCM prepares a new sub CA certificate and
acks.
# One follower stops receiving from the leader before the {{rotationCommit}}
request reaches it.
# The other SCMs commit the rotation. The leader takes a snapshot and purges
its log past the rotation requests.
# The follower is reachable again and installs the checkpoint.
The follower has to drop out between its ack and the commit and stay away until
the leader has purged. By code reading, with the default segment size (64MB)
and purge gap (1000000) that needs a long outage. Without log purging the
leader keeps the requests and the follower applies them from the log (code
reading, not run).
h3. Impact
Observed on a mini cluster, where Ratis runs without TLS. Wherever the follower
becomes leader below, the test moved leadership to it: with the admin command
after the polling interval, and with a Ratis leadership transfer inside it,
because the admin command is refused there.
* The follower keeps following the log, with its replicated state past the
rotation and its certificates on the old root. Until it is restarted it trusts
only the old root.
* A restart does not repair it. After one restart it also trusts the new root.
After a second restart {{SCMCertificateClient.getCACertificate}} returns the
new root, while its own certificate chain, the VERSION file and {{scm/sub-ca}}
are still on the old one.
* If it becomes leader inside {{hdds.x509.rootca.certificate.polling.interval}}
after the rotation (run after one restart), it terminates at leader ready with
"Root CA certificate ID 7 in stateful storage is bigger than current scm's root
CA certificate ID 1". Another SCM took over 10s later.
* If it becomes leader later (run without a restart; after one restart by code
reading, because {{getCACertificate}} still returns the old root then),
{{RootCARotationManager.MonitorTask}} sees the old root inside its grace period
and starts another rotation. All SCMs end on a third root and the root that was
just introduced is replaced. This is the only way the SCM recovers. The
rotation started at once in the test. By code reading it is scheduled for the
next {{hdds.x509.ca.rotation.time-of-day}} when the old root is still valid at
that time.
* If it becomes leader later and was restarted twice (run), no rotation starts.
It stays leader and signs with the sub CA certificate under the old root.
Argued, not run: the sub CA certificate of the follower expires with the old
root, at most twice {{hdds.x509.renew.grace.duration}} after the rotation, and
nothing but a rotation replaces an SCM sub CA certificate. With
{{hdds.grpc.tls.enabled=true}}, Ratis between SCMs and the checkpoint download
({{InterSCMGrpcClient}}, {{InterSCMGrpcProtocolService}}) both use the trust
manager of the certificate client and require a client certificate. A follower
that was cut off and still trusts only the old root can then neither accept
requests from the leader nor download the checkpoint, so step 5 of the trigger
needs a restart of the follower first, which makes it fetch the new root at
startup. That order was not run, with or without the patch.
By code reading, not run: the release tags ozone-1.4.0 to ozone-2.2.1 carry the
same {{rotationCommit}} and the same checkpoint install.
h3. Reproduction
PASS, unmodified source. [^TestBugMC6CheckpointSkipsRotation.java]
({{hadoop-ozone/integration-test}}) starts a secure mini cluster with three
SCMs. Certificate lifetimes are shortened so that the leader rotates the root
CA three minutes after SCM init, and log purging is enabled with a small
segment size, purge gap and snapshot threshold. The only intervention is a
partition: the Ratis test hooks of {{RaftServerImpl.appendEntries}} and
{{RaftServerImpl.installSnapshot}} reject what is sent to the first follower
that is offered the {{rotationCommit}} entry, until the leader has purged that
entry. After the follower has installed the checkpoint, the new root is in its
certificate store while its root certificate and VERSION file are the old ones
and {{scm/sub-ca-next}} is still present. A passing test means the defect is
present. It takes about three minutes.
h3. Patch
[^MC-6-complete-prepared-rotation-after-checkpoint-install.patch], against
b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
5108acff480.
After a checkpoint install, {{RootCARotationManager.completePreparedRotation}}
looks at the certificate prepared in {{scm/sub-ca-next}}. If its root is newer
than the current one and is present in the certificate store, the rotation to
it is committed, because the leader stores a new root there only after
{{rotationCommitted}}. The SCM then runs the commit steps locally, through a
{{RootCARotationHandlerImpl}} that is not wrapped in the Ratis proxy. Otherwise
nothing changes.
A failure while reading the prepared certificate or deciding is logged and
leaves the SCM as it is today. A failed commit step is not caught, so the SCM
never keeps running with its directories half switched. At this commit the
handler stops the SCM itself, as it does for a commit applied from the log
(run: no further step is executed and the directories are untouched; the stop
does not finish when it is called on this thread, which is the existing
behaviour of the handler, HDDS-16911, and not changed here). An exception from
the handler fails {{installCheckpoint}} and {{SCMStateMachine.reinitialize}},
Ratis closes the server and the SCM terminates (run with the handler changed to
throw).
Covered by {{testCompletePreparedRotation}} and
{{testCompletePreparedRotationThrowsFailedCommitStep}} in the existing
{{TestRootCARotationManager}}, for a prepared rotation that is not committed,
for one that is, and for a commit step that fails. The call from
{{installCheckpoint}} is exercised by the attached reproduction only, because
no existing suite combines a secure SCM HA cluster with a root CA rotation. The
reproduction with its last three assertions inverted can serve as an
integration test if one is wanted. With the patch the SCM HA and root CA
rotation unit suites of {{hadoop-hdds/server-scm}} (114 tests),
{{TestSCMInstallSnapshot}}, {{TestSCMInstallSnapshotWithHA}} and
{{TestSecretKeySnapshot}} pass and checkstyle is clean. The reproduction then
fails at the assertion that expects the old root: the follower logs "Rotation
to root certificate 7 is committed, switching to the prepared sub CA
certificate 10" during the install and ends on the new root with
{{scm/sub-ca-next}} gone. In a further run it became leader inside the polling
interval and kept running without starting a rotation.
Not covered by the patch: an SCM that is already in this state is not repaired
at startup. Nothing happens when the new root is not yet, or never, in the
certificate store of the installed checkpoint, that is a checkpoint taken
before the leader has stored it, or a leader that failed before storing it. A
follower that applied {{rotationCommit}} but not {{rotationCommitted}} before
it was cut off is not completed either.
Found by TLA+ model checking and code review of the SCM root CA rotation paths
under HDDS-15926, on commit b9b33851b20fa11baf4b07982de9afb8ea44f85a. Checked
against HDDS issues and apache/ozone pull requests for duplicates before
filing. The attached patch is a proposal for review. Generated with Specula
(Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
