[
https://issues.apache.org/jira/browse/ZOOKEEPER-5062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18092248#comment-18092248
]
Santosh Kumar Sahu commented on ZOOKEEPER-5062:
-----------------------------------------------
Hi [~andor] [~showuon]
Below is our analysis regarding this issue - Could you help in understanding
whether below commit fixes can help in avoiding the zk pod stuck issue?
ZOOKEEPER-4986 Disable reverse DNS lookup in TLS client and server - ASF Jira
ZOOKEEPER-4790 TLS Quorum hostname verification breaks in some scenarios - ASF
Jira
{code:java}
I saw some DNS related error logs in pod-0 during quorum formation
when pod1 or pod-2 is trying to connect to pod-0, pod-0 is not able to resolve
ip to fully qualified domain name of zk service
this issue comes only in the starting, then it stabilizes..
I suspect that coreDNS component is having delay while resolving ips when pod
is restarted and when new IP is assigned to pod
See logs - If you see below logs , 192.168.38.132 is of another zk member, but
this ip could have been resolved to
mss-bss-msg-msg2-zookeeper-1.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc but
it didnt happen, so ip is being checked in client cert's SAN field which
obviously does not include ip(which is volatile). And it rejects the connection
from peer zk member.
------------------------------------------------------------------------------------------------------------
{"version": "1.2.0", "timestamp": "2026-06-25T15:30:37.694+02:00", "severity":
"error", "service_id": "mss-data-messagebus-kf", "message": "Failed to verify
hostname: 192.168.38.132 javax.net.ssl.SSLPeerUnverifiedException: Certificate
for <192.168.38.132> doesn't match any of the subject alternative names:
[mss-bss-msg-msg2-zookeeper-client, mss-bss-msg-msg2-zookeeper-client.caf-n164,
mss-bss-msg-msg2-zookeeper-client.caf-n164.svc,
mss-bss-msg-msg2-zookeeper-client.caf-n164.svc.cluster.local,
mss-bss-msg-msg2-zookeeper-0.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc.cluster.local,
mss-bss-msg-msg2-zookeeper-0.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-client.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-client.caf-n164.svc.cluster.local,
*.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc.cluster.local]|\tat
org.apache.zookeeper.common.ZKHostnameVerifier.matchIPAddress(ZKHostnameVerifier.java:197)|\tat
org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:165)|\tat
org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189)|\tat
org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)|\tat
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkClientCerts(CertificateMessage.java:1216)|\tat
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1136)|\tat
java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1113)|\tat
----------------------------------------------------------------------------------------------------------------
Now see below logs -
If you will see below request , ip is resolved to something like ip.service
FQDN, DNS is not able to return proper FQDN.
there are many logs coming like this.
---------------------------------------------------------------------------------------------------------------
{"version": "1.2.0", "timestamp": "2026-06-25T15:29:32.978+02:00", "severity":
"error", "service_id": "mss-data-messagebus-kf", "message": "Failed to verify
hostname:
192-168-164-148.mss-bss-msg-msg2-zookeeper-jmx.caf-n164.svc.cluster.local
javax.net.ssl.SSLPeerUnverifiedException: Certificate for
<192-168-164-148.mss-bss-msg-msg2-zookeeper-jmx.caf-n164.svc.cluster.local>
doesn't match any of the subject alternative names:
[mss-bss-msg-msg2-zookeeper-client, mss-bss-msg-msg2-zookeeper-client.caf-n164,
mss-bss-msg-msg2-zookeeper-client.caf-n164.svc,
mss-bss-msg-msg2-zookeeper-client.caf-n164.svc.cluster.local,
mss-bss-msg-msg2-zookeeper-0.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc.cluster.local,
mss-bss-msg-msg2-zookeeper-0.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-client.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-client.caf-n164.svc.cluster.local,
*.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc,
*.mss-bss-msg-msg2-zookeeper-nodes.caf-n164.svc.cluster.local]|\tat
org.apache.zookeeper.common.ZKHostnameVerifier.matchDNSName(ZKHostnameVerifier.java:230)|\tat
org.apache.zookeeper.common.ZKHostnameVerifier.verify(ZKHostnameVerifier.java:171)|\tat
org.apache.zookeeper.common.ZKTrustManager.performHostVerification(ZKTrustManager.java:189)|\tat
org.apache.zookeeper.common.ZKTrustManager.checkClientTrusted(ZKTrustManager.java:93)|\tat
-------------------------------------------------------------------------
DNS issue in k8s, which causes zk pod to stuck in crashloopbackoff when it
fails to resolve to correct hostname in the first place.
And after this DNS issue is happened, zk-1 pod is not able to join the quorum,
it keeps printing "Oracle indicates not to follow" log and keeps doing leader
election and then cluster is not operational.
{code}
Thanks
Rahul
> Zookeeper cluster is non operational after DNS issue while zk pod starting
> --------------------------------------------------------------------------
>
> Key: ZOOKEEPER-5062
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-5062
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
> Affects Versions: 3.8.4
> Environment: Check below pod logs for all 3 pods
> [^zk-pod-0-logs.txt]
> [^zk-pod-1logs.txt]
> [^zk-pod-2logs.txt]
> Reporter: Santosh Kumar Sahu
> Priority: Blocker
> Attachments: zk-pod-0-logs.txt, zk-pod-1logs.txt, zk-pod-2logs.txt
>
>
> We have deployed zookeeper 3.8.4 version in k8s in the form of statefulset.
> We have 3 replicas and when we restart all pods simultaneously 3-4 times,
> zookeeper cluster is down and no request is served.
> Some DNS related logs get printed in one zk pod which keeps restarting
> Below is the pod status
> root@node-100-77-160-64:/home/mynamespace# kubectl get pod -n mynamespace -w
> | grep zoo
> cluster-p-zookeeper-0 1/1 Running
> 0 7m25s
> cluster-p-zookeeper-1 0/1 Running
> 5 (36s ago) 7m26s
> cluster-p-zookeeper-2 1/1 Running
> 0 7m26s
> cluster-p-zookeeper-bragent-75d88ffc5-nft5l 1/1 Running
> 0 22h
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
