zap51 commented on issue #7537: URL: https://github.com/apache/cloudstack/issues/7537#issuecomment-1656579422
@weizhouapache I tested this scenario with 3 control plane nodes. Guess `etcd` is refusing to work with a single node, perhaps issue with the quorum https://etcd.io/docs/v3.3/faq/#why-an-odd-number-of-cluster-members? 1. Started with 3 nodes, cluster works as expected. 2. Stopped one node, cluster works with 2 nodes as expected. 3. Stopped 2 nodes, cluster stops working with single node. 4. Start one more control plane node totalling 2, cluster starts to work. I also took a look at the VR's HAProxy under these conditions, active health checks are enabled on k8s api server 6443, backends are correctly marked down without any issues. Logged into master node where there was only single node active saw that k8s-api-server is continuously crashing. Also looks `etcd` is stuck in election storm ``` # crictl logs 100c1133f3192 WARN[0000] runtime connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead. ERRO[0000] unable to determine runtime API version: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory" {"level":"info","ts":"2023-07-29T06:48:38.292Z","caller":"etcdmain/etcd.go:73","msg":"Running: ","args":["etcd","--advertise-client-urls=https://10.1.1.4:2379","--cert-file=/etc/kubernetes/pki/etcd/server.crt","--client-cert-auth=true","--data-dir=/var/lib/etcd","--experimental-initial-corrupt-check=true","--initial-advertise-peer-urls=https://10.1.1.4:2380","--initial-cluster=cks-control-1898b776268=https://10.1.1.4:2380","--key-file=/etc/kubernetes/pki/etcd/server.key","--listen-client-urls=https://127.0.0.1:2379,https://10.1.1.4:2379","--listen-metrics-urls=http://127.0.0.1:2381","--listen-peer-urls=https://10.1.1.4:2380","--name=cks-control-1898b776268","--peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt","--peer-client-cert-auth=true","--peer-key-file=/etc/kubernetes/pki/etcd/peer.key","--peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt","--snapshot-count=10000","--trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt"]} {"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"etcdmain/etcd.go:116","msg":"server has been already initialized","data-dir":"/var/lib/etcd","dir-type":"member"} {"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:131","msg":"configuring peer listeners","listen-peer-urls":["https://10.1.1.4:2380"]} {"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:479","msg":"starting with peer TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/peer.crt, key = /etc/kubernetes/pki/etcd/peer.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]} {"level":"info","ts":"2023-07-29T06:48:38.293Z","caller":"embed/etcd.go:139","msg":"configuring client listeners","listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"]} {"level":"info","ts":"2023-07-29T06:48:38.294Z","caller":"embed/etcd.go:308","msg":"starting an etcd server","etcd-version":"3.5.3","git-sha":"0452feec7","go-version":"go1.16.15","go-os":"linux","go-arch":"amd64","max-cpu-set":4,"max-cpu-available":4,"member-initialized":true,"name":"cks-control-1898b776268","data-dir":"/var/lib/etcd","wal-dir":"","wal-dir-dedicated":"","member-dir":"/var/lib/etcd/member","force-new-cluster":false,"heartbeat-interval":"100ms","election-timeout":"1s","initial-election-tick-advance":true,"snapshot-count":10000,"snapshot-catchup-entries":5000,"initial-advertise-peer-urls":["https://10.1.1.4:2380"],"listen-peer-urls":["https://10.1.1.4:2380"],"advertise-client-urls":["https://10.1.1.4:2379"],"listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"],"cors":["*"],"host-whitelist":["*"],"initial-cluster":"","initial-cluster-state":"new","initial-cluster-token":"","quota-size-bytes":2147483648, "pre-vote":true,"initial-corrupt-check":true,"corrupt-check-time-interval":"0s","auto-compaction-mode":"periodic","auto-compaction-retention":"0s","auto-compaction-interval":"0s","discovery-url":"","discovery-proxy":"","downgrade-check-interval":"5s"} {"level":"info","ts":"2023-07-29T06:48:38.301Z","caller":"etcdserver/backend.go:81","msg":"opened backend db","path":"/var/lib/etcd/member/snap/db","took":"7.349991ms"} {"level":"info","ts":"2023-07-29T06:48:39.062Z","caller":"etcdserver/server.go:508","msg":"recovered v2 store from snapshot","snapshot-index":1600160,"snapshot-size":"13 kB"} {"level":"info","ts":"2023-07-29T06:48:39.062Z","caller":"etcdserver/server.go:521","msg":"recovered v3 backend from snapshot","backend-size-bytes":13955072,"backend-size":"14 MB","backend-size-in-use-bytes":4677632,"backend-size-in-use":"4.7 MB"} {"level":"info","ts":"2023-07-29T06:48:39.291Z","caller":"etcdserver/raft.go:483","msg":"restarting local member","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","commit-index":1606465} {"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d switched to configuration voters=(139226378462478669 394890042043021191 17681339031669231524)"} {"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became follower at term 11"} {"level":"info","ts":"2023-07-29T06:48:39.292Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"newRaft 1eea1a74f9ec14d [peers: [1eea1a74f9ec14d,57aee66a9503787,f560bc2b67e4cfa4], term: 11, commit: 1606465, applied: 1600160, lastindex: 1606469, lastterm: 11]"} {"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"api/capability.go:75","msg":"enabled capabilities for version","cluster-version":"3.5"} {"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"57aee66a9503787","recovered-remote-peer-urls":["https://10.1.1.151:2380"]} {"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"f560bc2b67e4cfa4","recovered-remote-peer-urls":["https://10.1.1.183:2380"]} {"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:278","msg":"recovered/added member from store","cluster-id":"eddf204d5d97ae54","local-member-id":"1eea1a74f9ec14d","recovered-remote-peer-id":"1eea1a74f9ec14d","recovered-remote-peer-urls":["https://10.1.1.4:2380"]} {"level":"info","ts":"2023-07-29T06:48:39.292Z","caller":"membership/cluster.go:287","msg":"set cluster version from store","cluster-version":"3.5"} {"level":"warn","ts":"2023-07-29T06:48:39.295Z","caller":"auth/store.go:1220","msg":"simple token is not cryptographically signed"} {"level":"info","ts":"2023-07-29T06:48:39.297Z","caller":"mvcc/kvstore.go:345","msg":"restored last compact revision","meta-bucket-name":"meta","meta-bucket-name-key":"finishedCompactRev","restored-compact-revision":1422383} {"level":"info","ts":"2023-07-29T06:48:39.308Z","caller":"mvcc/kvstore.go:415","msg":"kvstore restored","current-rev":1425171} {"level":"info","ts":"2023-07-29T06:48:39.312Z","caller":"etcdserver/quota.go:94","msg":"enabled backend quota with default value","quota-name":"v3-applier","quota-size-bytes":2147483648,"quota-size":"2.1 GB"} {"level":"info","ts":"2023-07-29T06:48:39.315Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.315Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.316Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.317Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787"} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"57aee66a9503787","remote-peer-urls":["https://10.1.1.151:2380"]} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/peer.go:133","msg":"starting remote peer","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.318Z","caller":"rafthttp/pipeline.go:72","msg":"started HTTP pipelining with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:169","msg":"started stream writer with remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/peer.go:137","msg":"started remote peer","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/transport.go:317","msg":"added remote peer","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4","remote-peer-urls":["https://10.1.1.183:2380"]} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"etcdserver/corrupt.go:46","msg":"starting initial corruption check","local-member-id":"1eea1a74f9ec14d","timeout":"7s"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream Message","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"info","ts":"2023-07-29T06:48:39.319Z","caller":"rafthttp/stream.go:395","msg":"started stream reader with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"1eea1a74f9ec14d","remote-peer-id":"f560bc2b67e4cfa4"} {"level":"warn","ts":"2023-07-29T06:48:40.977Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"1eea1a74f9ec14d","requested-revision":1425171,"remote-peer-endpoint":"https://10.1.1.151:2380","error":"Get \"https://10.1.1.151:2380/members/hashkv\": dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/corrupt.go:289","msg":"failed hash kv request","local-member-id":"1eea1a74f9ec14d","requested-revision":1425171,"remote-peer-endpoint":"https://10.1.1.183:2380","error":"Get \"https://10.1.1.183:2380/members/hashkv\": dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/corrupt.go:116","msg":"initial corruption checking passed; no corruption","local-member-id":"1eea1a74f9ec14d"} {"level":"info","ts":"2023-07-29T06:48:42.978Z","caller":"etcdserver/server.go:842","msg":"starting etcd server","local-member-id":"1eea1a74f9ec14d","local-server-version":"3.5.3","cluster-id":"eddf204d5d97ae54","cluster-version":"3.5"} {"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:688","msg":"starting with client TLS","tls-info":"cert = /etc/kubernetes/pki/etcd/server.crt, key = /etc/kubernetes/pki/etcd/server.key, client-cert=, client-key=, trusted-ca = /etc/kubernetes/pki/etcd/ca.crt, client-cert-auth = true, crl-file = ","cipher-suites":[]} {"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:277","msg":"now serving peer/client/metrics","local-member-id":"1eea1a74f9ec14d","initial-advertise-peer-urls":["https://10.1.1.4:2380"],"listen-peer-urls":["https://10.1.1.4:2380"],"advertise-client-urls":["https://10.1.1.4:2379"],"listen-client-urls":["https://10.1.1.4:2379","https://127.0.0.1:2379"],"listen-metrics-urls":["http://127.0.0.1:2381"]} {"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:763","msg":"serving metrics","address":"http://127.0.0.1:2381"} {"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"etcdserver/server.go:752","msg":"starting initial election tick advance","election-ticks":10} {"level":"info","ts":"2023-07-29T06:48:42.982Z","caller":"embed/etcd.go:581","msg":"serving peer traffic","address":"10.1.1.4:2380"} {"level":"info","ts":"2023-07-29T06:48:42.992Z","caller":"embed/etcd.go:553","msg":"cmux::serve","address":"10.1.1.4:2380"} {"level":"warn","ts":"2023-07-29T06:48:44.318Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:44.318Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:44.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:44.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:44.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:48:46.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:46.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:46.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:48.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:48:49.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:49.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:49.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:48:49.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:48:49.983Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:49.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:48:51.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:48:51.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:51.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:48:53.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:53.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:48:54.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:54.319Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:54.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:54.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:55.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:48:56.984Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:57.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:48:58.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:48:58.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:48:58.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:48:59.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:59.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:59.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:48:59.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:00.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:01.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:02.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:03.984Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:49:04.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:04.320Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:04.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:04.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:04.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:06.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:07.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:09.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:09.321Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:09.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:09.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:09.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:10.985Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:49:11.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:11.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:13.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:14.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:14.322Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:14.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:14.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:15.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:16.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:17.986Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:18.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:19.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:19.323Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:19.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:19.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:20.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:20.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:20.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:21.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:21.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:22.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:24.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:24.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:24.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:24.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:24.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:24.987Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:25.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:27.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:29.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:29.324Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:29.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:29.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:29.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:31.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:31.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:31.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"warn","ts":"2023-07-29T06:49:31.987Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:33.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:34.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:34.325Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:34.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:34.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:34.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:34.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:34.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:36.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:38.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:38.988Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:49:39.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:39.326Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:39.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:39.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:40.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:41.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:41.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:42.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:43.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:44.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:44.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:44.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:44.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:49:45.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:45.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:45.989Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:47.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:49.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:49.327Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:49.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:49.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:49.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:51.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:51.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:49:51.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"warn","ts":"2023-07-29T06:49:52.990Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:52.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:54.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:54.328Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:54.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:49:54.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:54.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:56.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:49:58.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:49:58.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:49:58.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:49:59.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:59.329Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:59.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:59.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:49:59.990Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:00.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:01.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:01.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:03.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:03.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:04.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:04.330Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:04.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:04.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:05.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:06.991Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:07.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:09.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:09.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:09.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:09.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:09.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:10.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:11.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:12.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:13.992Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:50:14.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:14.331Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:14.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:14.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:14.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:16.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:16.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:18.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:19.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:19.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:19.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:19.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:19.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:20.993Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:50:21.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:21.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:21.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:23.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:23.594Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:24.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:24.332Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:24.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:24.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:25.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:27.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:27.994Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:28.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:29.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:29.333Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:29.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:29.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:30.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:31.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:31.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:32.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:34.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:34.334Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:34.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:34.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:34.394Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:34.995Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:36.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:36.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:37.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:39.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:39.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:39.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:39.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:39.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:41.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:41.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:41.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:41.996Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:43.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:44.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:44.335Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:44.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:44.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:45.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:46.994Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:46.995Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:48.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:48.996Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"warn","ts":"2023-07-29T06:50:49.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:49.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:49.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:50:49.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:50.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:51.403Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:50:51.403Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:52.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:54.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:54.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:54.336Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:54.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:54.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:55.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:55.997Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:57.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:50:59.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:59.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:59.341Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:50:59.341Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:50:59.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:01.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:01.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:51:01.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"warn","ts":"2023-07-29T06:51:02.998Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:03.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:04.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:51:04.337Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:51:04.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"warn","ts":"2023-07-29T06:51:04.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: i/o timeout"} {"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:04.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:06.794Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:08.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:09.338Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:09.339Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:09.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:09.342Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:09.998Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:10.393Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:11.404Z","caller":"etcdhttp/metrics.go:173","msg":"serving /health false; no leader"} {"level":"warn","ts":"2023-07-29T06:51:11.404Z","caller":"etcdhttp/metrics.go:86","msg":"/health error","output":"{\"health\":\"false\",\"reason\":\"RAFT NO LEADER\"}","status-code":503} {"level":"info","ts":"2023-07-29T06:51:12.193Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:12.194Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:13.993Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:14.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:14.340Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"57aee66a9503787","rtt":"0s","error":"dial tcp 10.1.1.151:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:14.343Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"warn","ts":"2023-07-29T06:51:14.343Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"f560bc2b67e4cfa4","rtt":"0s","error":"dial tcp 10.1.1.183:2380: connect: no route to host"} {"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:15.793Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} {"level":"warn","ts":"2023-07-29T06:51:16.999Z","caller":"etcdserver/server.go:2063","msg":"failed to publish local member to cluster through raft","local-member-id":"1eea1a74f9ec14d","local-member-attributes":"{Name:cks-control-1898b776268 ClientURLs:[https://10.1.1.4:2379]}","request-path":"/0/members/1eea1a74f9ec14d/attributes","publish-timeout":"7s","error":"etcdserver: request timed out"} {"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d is starting a new election at term 11"} {"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d became pre-candidate at term 11"} {"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d received MsgPreVoteResp from 1eea1a74f9ec14d at term 11"} {"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to 57aee66a9503787 at term 11"} {"level":"info","ts":"2023-07-29T06:51:17.593Z","logger":"raft","caller":"etcdserver/zap_raft.go:77","msg":"1eea1a74f9ec14d [logterm: 11, index: 1606469] sent MsgPreVote request to f560bc2b67e4cfa4 at term 11"} ``` and from inside the container ``` sh-5.1# etcdctl --cluster=true endpoint health {"level":"warn","ts":"2023-07-29T06:55:21.108Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000432380/127.0.0.1:2379","attempt":0,"error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"} Error: failed to fetch endpoints from etcd cluster member list: context deadline exceeded ``` Could be the reason k8s-api-server not responding. Cheers, Jayanth -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
