You are probably running into ZOOKEEPER-4790 [1] here. When we encountered this back in the day [2] we figured out that enabling FIPS mode bypasses all the ZK specific TLS checks and makes it work. In the ZK version you are on it is not yet enabled by default, you could either update or set *zookeeper.fips-mode *and this error _should_ go away.
Good luck :) Best, Sönke [1] https://issues.apache.org/jira/browse/ZOOKEEPER-4790 [2] https://github.com/stackabletech/zookeeper-operator/issues/760 On Tue, Nov 19, 2024 at 9:08 AM Dharani (Jira) <j...@apache.org> wrote: > Dharani created ZOOKEEPER-4887: > ---------------------------------- > > Summary: Zookeeper quorum formation fails when TLS is enabled > in k8s env > Key: ZOOKEEPER-4887 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-4887 > Project: ZooKeeper > Issue Type: Bug > Affects Versions: 3.8.3 > Reporter: Dharani > > > We have three(3) node zookeeper cluster running as a pod on Kubernetes > cluster, zookeeper quorum formation fails with TLS handshake error, as the > server name in the https request does not match with any of the SANs in the > certificate configured for zookeeper server. Server name in the request is > of the form "x-x-x-x.kubernetes.default.svc.cluster.local" (where x-x-x-x > is the IP address of the POD), and I am unable to understand the reason > behind pre-pending FQDN with a IP address. > > > > Please find below the extract of the error logs from the zookeeper POD > {code:java} > [myid:] - ERROR [LearnerHandler-/192.168.220. > 10:46516:o.a.z.c.ZKTrustManager@191] - Failed to verify host address: > 192.168.220.10 > javax.net.ssl.SSLPeerUnverifiedException: Certificate for <192.168.220.10> > doesn't match any of the subject alternative names: > [eric-data-coordinator-zk, eric-data-coordinator-zk.zdhagxx1, > eric-data-coordinator-zk.zdhagxx1.svc, > eric-data-coordinator-zk.zdhagxx1.svc.cluster.local, > *.eric-data-coordinator-zk-ensemble-service.zdhagxx1.svc.cluster.local, > certified-scrape-target] > > org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197) > > org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165) > > org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:180) > > org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93) > > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1285) > > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) > > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) > java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) > > java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) > > java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) > > java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) > java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) > java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511) > > java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) > > java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) > > java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:926) > java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:372) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172) > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699) > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693) > java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) > java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271) > java.base/java.io.DataInputStream.readInt(DataInputStream.java:392) > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:96) > > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86) > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:134) > org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:472)[myid:] > - ERROR [LearnerHandler-/192.168.220.10:46516:o.a.z.c.ZKTrustManager@192] > - Failed to verify hostname: > 192-168-220-10.eric-data-coordinator-zk.zdhagxx1.svc.cluster.local > javax.net.ssl.SSLPeerUnverifiedException: Certificate for > <192-168-220-10.eric-data-coordinator-zk.zdhagxx1.svc.cluster.local> > doesn't match any of the subject alternative names: > [eric-data-coordinator-zk, eric-data-coordinator-zk.zdhagxx1, > eric-data-coordinator-zk.zdhagxx1.svc, > eric-data-coordinator-zk.zdhagxx1.svc.cluster.local, > *.eric-data-coordinator-zk-ensemble-service.zdhagxx1.svc.cluster.local, > certified-scrape-target] > org.apache.zookeeper.common.ZKHostnameVerifier.matchDNSName(ZKHostnameVerifier.java:230) > org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:171) > > org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189) > > org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93) > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1285) > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1204) > java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) > java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) > > java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443) > > java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) > > java.base/sun.security.ssl.TransportContext.dispatch(TransportContext.java:183) > java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) > java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1511) > > java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421) > > java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456) > > java.base/sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:926) > java.base/sun.security.ssl.SSLSocketImpl.getSession(SSLSocketImpl.java:372) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.detectMode(UnifiedServerSocket.java:269) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.getSocket(UnifiedServerSocket.java:298) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedSocket.access$400(UnifiedServerSocket.java:172) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.getRealInputStream(UnifiedServerSocket.java:699) > > org.apache.zookeeper.server.quorum.UnifiedServerSocket$UnifiedInputStream.read(UnifiedServerSocket.java:693) > java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:252) > java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:271) > java.base/java.io.DataInputStream.readInt(DataInputStream.java:392) > org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:96) > > org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:86) > org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:134) > org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:472) > {code} > > > > -- > This message was sent by Atlassian Jira > (v8.20.10#820010) >
