Nick created ZOOKEEPER-5096:
-------------------------------
Summary: Quorum TLS fails to form a quorum in 3.9.6 (fips-enabled,
default)
Key: ZOOKEEPER-5096
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5096
Project: ZooKeeper
Issue Type: Bug
Components: security
Affects Versions: 3.9.6
Environment: Observed on Temurin 21.0.12.1+1.
Reporter: Nick
A recent commit
[8b4266550|https://github.com/apache/zookeeper/commit/8b4266550dbac68fd27586c023bbaa714f9f0754]
for fixing a CVE:
"Quorum TLS in FIPS mode accepts hostname-mismatched peer certificates", added
{{SSLParameters.setEndpointIdentificationAlgorithm("HTTPS")}} to _both_
directions of
quorum TLS in FIPS mode.
SunJSSE (X509TrustManagerImpl) only matches a _client_ certificate against the
peer IP (no reverse DNS lookup to check DNS Subject/SANs like ZKTrustManager
does for non-FIPS mode).
We have client certificates that are issued before the IP is known (in
Kubernetes), and so we can only use DNS names for verification.
Without additional logging, it appears like this:
On the accepting peer (the quorum TLS server):
{noformat}
INFO o.a.z.s.q.QuorumCnxManager$Listener$ListenerHandler@1073 - Received
connection request from /10.0.0.15:33280
INFO o.a.z.s.q.UnifiedServerSocket$UnifiedSocket@266 - Accepted TLS connection
from /10.0.0.15:33280 - NONE - SSL_NULL_WITH_NULL_NULL
WARN o.a.z.s.q.QuorumCnxManager@627 - Exception reading or writing challenge
java.net.SocketException: Socket is closed
at java.base/sun.security.ssl.SSLSocketImpl.getInputStream(Unknown Source)
at
o.a.z.s.q.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699)
at o.a.z.s.q.QuorumCnxManager.handleConnection(QuorumCnxManager.java:599)
{noformat}
on the client (quorum TLS client):
{noformat}
WARN o.a.z.s.q.QuorumCnxManager@398 - Cannot open channel to 10 at election
address <peer>:3888
javax.net.ssl.SSLHandshakeException: (certificate_unknown) Received fatal
alert: certificate_unknown
at o.a.z.s.q.QuorumCnxManager.initiateConnection(QuorumCnxManager.java:384)
{noformat}
It is a little more obvious when run with {{-Djavax.net.debug=ssl:handshake}}
{noformat}
javax.net.ssl|ERROR|...|Fatal (CERTIFICATE_UNKNOWN): Endpoint Identification
Algorithm HTTPS is not supported on the server side
"throwable" : {
java.security.cert.CertificateException: Endpoint Identification Algorithm
HTTPS is not supported on the server side
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(Unknown
Source)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(Unknown
Source)
at
java.base/sun.security.ssl.X509TrustManagerImpl.checkClientTrusted(Unknown
Source)
at
java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkClientCerts(Unknown
Source)
at
java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(Unknown
Source)
{noformat}
and we then see no IP SANs to match on:
{noformat}
ObjectId: 2.5.29.17 Criticality=true
SubjectAlternativeName [
DNSName: <sts>-headless.<namespace>.svc.cluster.local
DNSName: <sts>-0.<sts>-headless.<namespace>.svc.cluster.local
]
{noformat}
[HostnameChecker.matchIP|https://github.com/openjdk/jdk21u/blob/master/src/java.base/share/classes/sun/security/util/HostnameChecker.java]
only compares against {{ALTNAME_IP}} SANs and never DNS SANs.
h2. Current workaround
Revert back to pre-3.9.6 behaviour of no checking for the client certificate
(server name verification will still be performed).
{noformat}
ssl.quorum.clientHostnameVerification=false
{noformat}
or disabling {{fips-enabled=false}}, though this was causing me other issues
with metric scraping (I didn't dive deep into that yet).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
