[
https://issues.apache.org/jira/browse/HDDS-16035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov updated HDDS-16035:
-----------------------------------
Status: Patch Available (was: Open)
> Secure SCM HA cluster cannot start: the primary SCM refuses to sign
> certificates for the other SCMs when there is no leader
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16035
> URL: https://issues.apache.org/jira/browse/HDDS-16035
> Project: Apache Ozone
> Issue Type: Bug
> Components: SCM HA, Security
> Affects Versions: 2.2.0, 1.4.1
> Reporter: Sergey Soldatov
> Assignee: Sergey Soldatov
> Priority: Major
> Labels: pull-request-available
>
> I hit this while turning on security on a 3 node SCM HA cluster that had been
> running without it. SCM was unable to build the quorum. The only recovery was
> deleting the SCM data, which is not acceptable in production. The same
> symptom is reported by another user in apache/ozone discussion #7616 (Ozone
> 1.4.1 on Kubernetes with Kerberos, no vendor tooling involved).
> RCA:
> In a secure SCM HA cluster, an SCM cannot start until it has a sub CA
> certificate, and it asks the primary SCM for one over SCMSecurityProtocol.
> SCMSecurityProtocolServerSideTranslatorPB.submitRequest() rejects every
> request on that protocol unless the SCM serving it is the Ratis leader. So
> the other SCMs wait for a certificate, and the certificate waits for a
> leader that cannot be elected until those SCMs are up. Nothing breaks the
> loop, and the cluster never starts. The primary is running and reachable the
> whole time: it signed its own certificate without any leader, and it holds
> the root CA, but it answers every request with ServerNotLeaderException
> ("Could not determine the leader node"). Ozone's SCM HA documentation says
> only the primordial SCM can issue certificates for the other SCMs, so the
> leader check does not fit what this call does. Before 1.4.0 the same method
> had an explicit exemption for GetSCMCertificate; HDDS-8286 removed it so that
> a new SCM joining a healthy cluster is signed by whichever node leads, and
> bootstrap then inherited that rule.
> Suggested fix:
> Let a single narrow case skip the leader check: a GetSCMCertificate request
> that is not a renewal, served by the primary SCM (the one holding the root
> CA), at a moment when no leader is known. Signing on that path also needs a
> serial number and a database write, which normally go through Ratis, so it
> would use a direct serial allocator next to the existing
> SequenceIdGenerator.upgradeToCertificateSequenceId() and the existing
> unreplicated storeValidScmCertificate() write that persistPrimarySCMCerts()
> already uses on every primary start. The current checkValidCertID() check
> stays, so a repeated serial is refused and never issued, and the path logs a
> warning so an operator can see it ran. Everything else keeps the leader
> check: renewals, non primary SCMs, and OM and datanode certificates. When a
> leader exists nothing changes, so this is not a revert of HDDS-8286. It needs
> no proto change, no new config key, and no interface change. One known gap to
> note: a certificate signed this way is written only in the primary's local
> database, which is the same exposure persistPrimarySCMCerts() already has,
> and each SCM keeps its own certificate on local disk, so startup does not
> depend on the replicated copy.
> [~Sammi] I would really appreciate your feedback about the suggested fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
