Sergey Soldatov created HDDS-16035:
--------------------------------------
Summary: Secure SCM HA cluster cannot start: the primary SCM
refuses to sign certificates for the other SCMs when there is no leader
Key: HDDS-16035
URL: https://issues.apache.org/jira/browse/HDDS-16035
Project: Apache Ozone
Issue Type: Bug
Components: SCM HA, Security
Affects Versions: 2.2.0, 1.4.1
Reporter: Sergey Soldatov
Assignee: Sergey Soldatov
I hit this while turning on security on a 3 node SCM HA cluster that had been
running without it. SCM was unable to build the quorum. The only recovery was
deleting the SCM data, which is not acceptable in production. The same symptom
is reported by another user in apache/ozone discussion #7616 (Ozone 1.4.1 on
Kubernetes with Kerberos, no vendor tooling involved).
RCA:
In a secure SCM HA cluster, an SCM cannot start until it has a sub CA
certificate, and it asks the primary SCM for one over SCMSecurityProtocol.
SCMSecurityProtocolServerSideTranslatorPB.submitRequest() rejects every request
on that protocol unless the SCM serving it is the Ratis leader. So the other
SCMs wait for a certificate, and the certificate waits for a leader that
cannot be elected until those SCMs are up. Nothing breaks the loop, and the
cluster never starts. The primary is running and reachable the whole time: it
signed its own certificate without any leader, and it holds the root CA, but it
answers every request with ServerNotLeaderException ("Could not determine the
leader node"). Ozone's SCM HA documentation says only the primordial SCM can
issue certificates for the other SCMs, so the leader check does not fit what
this call does. Before 1.4.0 the same method had an explicit exemption for
GetSCMCertificate; HDDS-8286 removed it so that a new SCM joining a healthy
cluster is signed by whichever node leads, and bootstrap then inherited that
rule.
Suggested fix:
Let a single narrow case skip the leader check: a GetSCMCertificate request
that is not a renewal, served by the primary SCM (the one holding the root CA),
at a moment when no leader is known. Signing on that path also needs a serial
number and a database write, which normally go through Ratis, so it would use a
direct serial allocator next to the existing
SequenceIdGenerator.upgradeToCertificateSequenceId() and the existing
unreplicated storeValidScmCertificate() write that persistPrimarySCMCerts()
already uses on every primary start. The current checkValidCertID() check
stays, so a repeated serial is refused and never issued, and the path logs a
warning so an operator can see it ran. Everything else keeps the leader check:
renewals, non primary SCMs, and OM and datanode certificates. When a leader
exists nothing changes, so this is not a revert of HDDS-8286. It needs no proto
change, no new config key, and no interface change. One known gap to note: a
certificate signed this way is written only in the primary's local database,
which is the same exposure persistPrimarySCMCerts() already has, and each SCM
keeps its own certificate on local disk, so startup does not depend on the
replicated copy.
[~Sammi] I would really appreciate your feedback about the suggested fix.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
