Hi,
We using jmx-exporter <https://github.com/prometheus/jmx_exporter> to scrape
JMX Bean from zookeeper pod(deployed on Kubernetes) and export prometheus
metrics. Recently, we found a abnormal metric came out in production, as below
zookeeper_InMemoryDatatree_NodeCount{memberType=“Leader”, replicateId=“201" }
218.0
zookeeper_InMemoryDatatree_NodeCount{memberType=“Follower”, replicateId=“201"
} 218.0
zookeeper_InMemoryDataTree_NodeCount metric have two sample with different
memberType, but the pod should be Leader not Follower! And, we found
zookeeper_InMemoryDataTree_WatchCount metric have the same issue, meanwhile the
rest of metrics have not.
We tried to reproduce this by restarting a Leader Pod, make another pod to be
leader, but got bad luck.
As we dived in source code, we thought it may caused by DataTreeBean
duplicated, but we don’t have any evidences. I just wonder if anyone came into
this issue before, or if there is chance this is a bug.
Any advices would be helpful, thanks in advanced.
