Siyao Meng created HDDS-16936:
---------------------------------
Summary: SCM issues SCM role certificates without the SCM admin
check and copies the CA basic constraint for every role
Key: HDDS-16936
URL: https://issues.apache.org/jira/browse/HDDS-16936
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-require-scm-admin-for-scm-role-certificates.patch
h3. Mechanism
{{SCMSecurityProtocolServer.getEncodedCertToString}} signs every CSR for the
node type the request names. For {{NodeType.SCM}} it uses the root CA server
when the SCM holds one and the sub CA server otherwise, and
{{DefaultCAServer.getDuration}} gives that role {{hdds.x509.max.duration}}; the
result is an SCM sub CA certificate, stored with role SCM.
Two RPCs reach it with that role.
{{SCMSecurityProtocolServer.getSCMCertificate}} compares the cluster id of the
request with its own and calls {{ScmUtils.checkIfCertSignRequestAllowed}} (the
root CA rotation window). {{SCMSecurityProtocolServer.getCertificate}} (the
{{NodeDetailsProto}} form, added for Recon in HDDS-6212) forwards
{{nodeDetails.getNodeType()}} and checks nothing else. Neither consults
{{StorageContainerManager.checkAdminAccess}}; in this server only
{{removeExpiredCertificates}} does, while {{SCMBlockProtocolServer.addSCM}},
the other operation an SCM peer performs when it joins, does.
Separately, the sub CA server that issues OM, datanode and Recon certificates
is built with {{DefaultCAProfile}} ({{StorageContainerManager}}, HDDS-5052),
whose extension map accepts the basic constraints extension, and
{{DefaultApprover.sign}} copies every extension the profile lists from the CSR
into the certificate. So a basic constraints extension in a CSR is copied into
the certificate for every role, not only SCM.
{{DefaultCAServer.requestCertificate}} also completes its future exceptionally
when {{DefaultApprover.inspectCSR}} rejects the CSR and then continues into
{{signAndStoreCertificate}}, so a rejected CSR is still signed and persisted.
h3. Trigger
# An SCM role signing request ({{GetSCMCertificate}}, or {{GetCertificate}}
with node type {{SCM}}) whose remote user is not in the SCM admin set.
# A CSR for a role other than SCM that carries the basic constraints extension,
handled by a {{DefaultCAServer}} built with {{DefaultCAProfile}}.
# A CSR that {{DefaultApprover.inspectCSR}} rejects.
By history, not run on older code: {{getSCMCertificate}} and the
{{DefaultCAProfile}} sub CA arrived with HDDS-4861 and HDDS-5052 (1.2.0), the
{{NodeDetailsProto}} form with HDDS-6212 (1.3.0). The three core files are byte
identical between the pin and the 2.2.1 tag.
h3. Impact
* The SCM admin check that gates the other SCM peer operation ({{addSCM}}) does
not gate the issuance of SCM role certificates, which are CA certificates under
the cluster root with the maximum duration, and the CA basic constraint is not
limited to the SCM role.
* A CSR the approver rejected is still persisted in the SCM certificate store
and returned by {{listCertificate}}.
h3. Reproduction
The regression tests in the attached patch cover it and fail on unmodified
source. There is no standalone reproduction.
* {{TestDefaultCAServer.testBasicConstraintsAllowedForScmRoleOnly}}
({{hadoop-hdds/framework}}): a basic constraints CSR for the OM role is
rejected and not stored, the same CSR for the SCM role is issued with the
constraint and stored, and a CSR the profile rejects is neither signed nor
stored ({{MockCAStore}} now records what is stored). Without the change it
fails with {{Expected java.util.concurrent.ExecutionException to be thrown, but
nothing was thrown}}.
*
{{TestSecureOzoneCluster.assertScmSecurityProtocolRequiresAdminForScmRoleCertificate}}
({{hadoop-ozone/integration-test}}, inside
{{testSecureScmAndOmStartupAndAccessControl}} next to the existing SCM admin
protocol checks): both SCM role signing RPCs answer {{Access denied}} for a
remote user outside the SCM admin set. Without the change it fails with
{{Expected java.io.IOException to be thrown, but nothing was thrown}}.
h3. Patch
[^CR-3-require-scm-admin-for-scm-role-certificates.patch], against
b9b33851b20fa11baf4b07982de9afb8ea44f85a. It also applies to master at
084c60d4fb9.
{{SCMSecurityProtocolServer.getEncodedCertToString}} calls {{checkAdminAccess}}
with the RPC remote user when the role is SCM, so both RPCs are covered in one
place. {{DefaultCAServer.requestCertificate}} rejects a CSR that carries the
basic constraints extension for any role other than SCM before signing, and
returns without signing when the approver rejects the CSR. {{MockCAStore}}
records stored certificates for the test.
No wire or protocol change and no configuration change. A bootstrapping or
renewing SCM requests its certificate under its own login user
({{SCMCertificateClient}}), and the SCM admin set always contains the starter
user plus {{ozone.administrators}}, the same membership {{addSCM}} already
requires of it, so no new deployment requirement. OM, datanode and Recon CSRs
never carry the basic constraints extension (only {{SCMCertificateClient}} sets
it), so the second change rejects no legitimate CSR. In a mixed version cluster
an older SCM against a patched leader passes the same check it already passes
for {{addSCM}}, and a patched SCM against an older primary meets no new check.
Outside RPC, with no remote user, the check is a no op, as for
{{removeExpiredCertificates}}; with {{ozone.authorization.enabled}} false it
returns early, as every SCM admin check does.
With the patch {{TestDefaultCAServer}}, {{TestDefaultProfile}}
({{hadoop-hdds/framework}}) and {{TestRootCARotationManager}}
({{hadoop-hdds/server-scm}}) pass, the SCM part of
{{TestSecureOzoneCluster.testSecureScmAndOmStartupAndAccessControl}} passes,
and checkstyle on {{hadoop-hdds/framework}}, {{hadoop-hdds/server-scm}} and
{{hadoop-ozone/integration-test}} is clean. Not run: a secure multi SCM
bootstrap or certificate renewal with the check in place ({{TestSecretKeysApi}}
did not start on the test machine for a host name resolution reason unrelated
to the change), and the OM part of
{{testSecureScmAndOmStartupAndAccessControl}} times out on that machine with
and without the patch for the same reason. That bootstrap and renewal keep
working follows from the shared login user by code reading and should be
confirmed on CI before merge.
Found by code review of the SCM root CA rotation paths, as part of the TLA+
verification effort under HDDS-15926, on commit
b9b33851b20fa11baf4b07982de9afb8ea44f85a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Fable 5.1).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
