[
https://issues.apache.org/jira/browse/HDDS-16915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16915:
------------------------------
Attachment: MC-6-complete-prepared-rotation-after-checkpoint-install.patch
> SCM follower that installs a checkpoint taken after a root CA rotation commit
> stays on the old root certificate
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16915
> URL: https://issues.apache.org/jira/browse/HDDS-16915
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-6-complete-prepared-rotation-after-checkpoint-install.patch,
> TestBugMC6CheckpointSkipsRotation.java
>
>
> h3. Mechanism
> {{RootCARotationHandlerImpl.rotationCommit}} and {{rotationCommitted}} are
> replicated requests, but what they change is local to each SCM: they move
> {{scm/sub-ca-next}} to {{scm/sub-ca}}, write the new certificate ID to the
> VERSION file and reload the certificate client. None of this is in the SCM
> DB. When a follower catches up by installing a checkpoint,
> {{SCMHAManagerImpl.installCheckpoint}} replaces the DB and reloads the DB
> backed managers. The applied index moves past the two requests and the
> follower never applies them. It keeps its sub CA certificate under the old
> root, and the certificate it prepared for the new root stays in
> {{scm/sub-ca-next}}.
> h3. Trigger
> # Secure SCM HA with {{hdds.x509.ca.rotation.enabled=true}} and
> {{ozone.scm.ha.ratis.log.purge.enabled=true}} (both false by default).
> # A root CA rotation starts. Every SCM prepares a new sub CA certificate and
> acks.
> # One follower stops receiving from the leader before the {{rotationCommit}}
> request reaches it.
> # The other SCMs commit the rotation. The leader takes a snapshot and purges
> its log past the rotation requests.
> # The follower is reachable again and installs the checkpoint.
> The follower has to drop out between its ack and the commit and stay away
> until the leader has purged. By code reading, with the default segment size
> (64MB) and purge gap (1000000) that needs a long outage. Without log purging
> the leader keeps the requests and the follower applies them from the log
> (code reading, not run).
> h3. Impact
> Observed on a mini cluster, where Ratis runs without TLS. Wherever the
> follower becomes leader below, the test moved leadership to it: with the
> admin command after the polling interval, and with a Ratis leadership
> transfer inside it, because the admin command is refused there.
> * The follower keeps following the log, with its replicated state past the
> rotation and its certificates on the old root. Until it is restarted it
> trusts only the old root.
> * A restart does not repair it. After one restart it also trusts the new
> root. After a second restart {{SCMCertificateClient.getCACertificate}}
> returns the new root, while its own certificate chain, the VERSION file and
> {{scm/sub-ca}} are still on the old one.
> * If it becomes leader inside
> {{hdds.x509.rootca.certificate.polling.interval}} after the rotation (run
> after one restart), it terminates at leader ready with "Root CA certificate
> ID 7 in stateful storage is bigger than current scm's root CA certificate ID
> 1". Another SCM took over 10s later.
> * If it becomes leader later (run without a restart; after one restart by
> code reading, because {{getCACertificate}} still returns the old root then),
> {{RootCARotationManager.MonitorTask}} sees the old root inside its grace
> period and starts another rotation. All SCMs end on a third root and the root
> that was just introduced is replaced. This is the only way the SCM recovers.
> The rotation started at once in the test. By code reading it is scheduled for
> the next {{hdds.x509.ca.rotation.time-of-day}} when the old root is still
> valid at that time.
> * If it becomes leader later and was restarted twice (run), no rotation
> starts. It stays leader and signs with the sub CA certificate under the old
> root.
> Argued, not run: the sub CA certificate of the follower expires with the old
> root, at most twice {{hdds.x509.renew.grace.duration}} after the rotation,
> and nothing but a rotation replaces an SCM sub CA certificate. With
> {{hdds.grpc.tls.enabled=true}}, Ratis between SCMs and the checkpoint
> download ({{InterSCMGrpcClient}}, {{InterSCMGrpcProtocolService}}) both use
> the trust manager of the certificate client and require a client certificate.
> A follower that was cut off and still trusts only the old root can then
> neither accept requests from the leader nor download the checkpoint, so step
> 5 of the trigger needs a restart of the follower first, which makes it fetch
> the new root at startup. That order was not run, with or without the patch.
> By code reading, not run: the release tags ozone-1.4.0 to ozone-2.2.1 carry
> the same {{rotationCommit}} and the same checkpoint install.
> h3. Reproduction
> PASS, unmodified source. [^TestBugMC6CheckpointSkipsRotation.java]
> ({{hadoop-ozone/integration-test}}) starts a secure mini cluster with three
> SCMs. Certificate lifetimes are shortened so that the leader rotates the root
> CA three minutes after SCM init, and log purging is enabled with a small
> segment size, purge gap and snapshot threshold. The only intervention is a
> partition: the Ratis test hooks of {{RaftServerImpl.appendEntries}} and
> {{RaftServerImpl.installSnapshot}} reject what is sent to the first follower
> that is offered the {{rotationCommit}} entry, until the leader has purged
> that entry. After the follower has installed the checkpoint, the new root is
> in its certificate store while its root certificate and VERSION file are the
> old ones and {{scm/sub-ca-next}} is still present. A passing test means the
> defect is present. It takes about three minutes.
> h3. Patch
> [^MC-6-complete-prepared-rotation-after-checkpoint-install.patch], against
> b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
> 5108acff480.
> After a checkpoint install,
> {{RootCARotationManager.completePreparedRotation}} looks at the certificate
> prepared in {{scm/sub-ca-next}}. If its root is newer than the current one
> and is present in the certificate store, the rotation to it is committed,
> because the leader stores a new root there only after {{rotationCommitted}}.
> The SCM then runs the commit steps locally, through a
> {{RootCARotationHandlerImpl}} that is not wrapped in the Ratis proxy.
> Otherwise nothing changes.
> A failure while reading the prepared certificate or deciding is logged and
> leaves the SCM as it is today. A failed commit step is not caught, so the SCM
> never keeps running with its directories half switched. At this commit the
> handler stops the SCM itself, as it does for a commit applied from the log
> (run: no further step is executed and the directories are untouched; the stop
> does not finish when it is called on this thread, which is the existing
> behaviour of the handler, HDDS-16911, and not changed here). An exception
> from the handler fails {{installCheckpoint}} and
> {{SCMStateMachine.reinitialize}}, Ratis closes the server and the SCM
> terminates (run with the handler changed to throw).
> Covered by {{testCompletePreparedRotation}} and
> {{testCompletePreparedRotationThrowsFailedCommitStep}} in the existing
> {{TestRootCARotationManager}}, for a prepared rotation that is not committed,
> for one that is, and for a commit step that fails. The call from
> {{installCheckpoint}} is exercised by the attached reproduction only, because
> no existing suite combines a secure SCM HA cluster with a root CA rotation.
> The reproduction with its last three assertions inverted can serve as an
> integration test if one is wanted. With the patch the SCM HA and root CA
> rotation unit suites of {{hadoop-hdds/server-scm}} (114 tests),
> {{TestSCMInstallSnapshot}}, {{TestSCMInstallSnapshotWithHA}} and
> {{TestSecretKeySnapshot}} pass and checkstyle is clean. The reproduction then
> fails at the assertion that expects the old root: the follower logs "Rotation
> to root certificate 7 is committed, switching to the prepared sub CA
> certificate 10" during the install and ends on the new root with
> {{scm/sub-ca-next}} gone. In a further run it became leader inside the
> polling interval and kept running without starting a rotation.
> Not covered by the patch: an SCM that is already in this state is not
> repaired at startup. Nothing happens when the new root is not yet, or never,
> in the certificate store of the installed checkpoint, that is a checkpoint
> taken before the leader has stored it, or a leader that failed before storing
> it. A follower that applied {{rotationCommit}} but not {{rotationCommitted}}
> before it was cut off is not completed either.
> Found by TLA+ model checking and code review of the SCM root CA rotation
> paths under HDDS-15926, on commit b9b33851b20fa11baf4b07982de9afb8ea44f85a.
> Checked against HDDS issues and apache/ozone pull requests for duplicates
> before filing. The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
