gerry-vip opened a new issue #5862: URL: https://github.com/apache/apisix/issues/5862
### Issue description # Apisix version  # Apisix pod status  # Etcd detail log  `[root@localhost opt]# kubectl logs -n apisix apisix-etcd-1 etcd 09:51:17.03 etcd 09:51:17.04 Welcome to the Bitnami etcd container etcd 09:51:17.04 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd etcd 09:51:17.04 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues etcd 09:51:17.04 etcd 09:51:17.05 INFO ==> ** Starting etcd setup ** etcd 09:51:17.06 INFO ==> Validating settings in ETCD_* env vars.. etcd 09:51:17.06 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment. etcd 09:51:17.06 INFO ==> Initializing etcd etcd 09:51:17.08 INFO ==> Detected data from previous deployments etcd 09:51:39.31 WARN ==> Cluster not responding! etcd 09:51:39.31 WARN ==> Disaster recovery is disabled, the cluster will try to recover on it's own etcd 09:51:39.32 INFO ==> ** etcd setup finished! ** etcd 09:51:39.33 INFO ==> ** Starting etcd ** 2021-12-20 09:51:39.350386 I | pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2379 2021-12-20 09:51:39.350443 I | pkg/flags: recognized and used environment variable ETCD_AUTO_TLS=false 2021-12-20 09:51:39.350451 I | pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false 2021-12-20 09:51:39.350458 I | pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data 2021-12-20 09:51:39.350479 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380 2021-12-20 09:51:39.350486 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380 2021-12-20 09:51:39.350492 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new 2021-12-20 09:51:39.350496 I | pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s 2021-12-20 09:51:39.350504 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 2021-12-20 09:51:39.350509 I | pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 2021-12-20 09:51:39.350513 I | pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=info 2021-12-20 09:51:39.350521 I | pkg/flags: recognized and used environment variable ETCD_NAME=apisix-etcd-1 2021-12-20 09:51:39.350525 I | pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false 2021-12-20 09:51:39.350585 W | pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE= 2021-12-20 09:51:39.350595 W | pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes 2021-12-20 09:51:39.350598 W | pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots 2021-12-20 09:51:39.350601 W | pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin 2021-12-20 09:51:39.350604 W | pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd 2021-12-20 09:51:39.350608 W | pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD= 2021-12-20 09:51:39.350611 W | pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.apisix.svc.cluster.local 2021-12-20 09:51:39.350617 W | pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no 2021-12-20 09:51:39.350620 W | pkg/flags: unrecognized environment variable ETCD_KEY_FILE= 2021-12-20 09:51:39.350623 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd 2021-12-20 09:51:39.350626 W | pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=no 2021-12-20 09:51:39.350629 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOT_FILENAME= 2021-12-20 09:51:39.350632 W | pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot 2021-12-20 09:51:39.350636 W | pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd 2021-12-20 09:51:39.350640 W | pkg/flags: unrecognized environment variable ETCD_CERT_FILE= 2021-12-20 09:51:39.350643 W | pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs 2021-12-20 09:51:39.350646 W | pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2021-12-20 09:51:39.350686 I | etcdmain: etcd Version: 3.4.16 2021-12-20 09:51:39.350691 I | etcdmain: Git SHA: d19fbe541 2021-12-20 09:51:39.350694 I | etcdmain: Go Version: go1.12.17 2021-12-20 09:51:39.350696 I | etcdmain: Go OS/Arch: linux/amd64 2021-12-20 09:51:39.350699 I | etcdmain: setting maximum number of CPUs to 2, total number of available CPUs is 2 2021-12-20 09:51:39.359832 N | etcdmain: the server is already initialized as member before, starting as etcd member... [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead 2021-12-20 09:51:39.361123 I | embed: name = apisix-etcd-1 2021-12-20 09:51:39.361133 I | embed: data dir = /bitnami/etcd/data 2021-12-20 09:51:39.361137 I | embed: member dir = /bitnami/etcd/data/member 2021-12-20 09:51:39.361140 I | embed: heartbeat = 100ms 2021-12-20 09:51:39.361145 I | embed: election = 1000ms 2021-12-20 09:51:39.361147 I | embed: snapshot count = 100000 2021-12-20 09:51:39.361158 I | embed: advertise client URLs = http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2379 2021-12-20 09:51:39.361162 I | embed: initial advertise peer URLs = http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380 2021-12-20 09:51:39.361166 I | embed: initial cluster = 2021-12-20 09:51:39.380214 I | etcdserver: restarting member 2c16fb63879f0d98 in cluster b0d7015fda1525c8 at commit index 3 raft2021/12/20 09:51:39 INFO: 2c16fb63879f0d98 switched to configuration voters=() raft2021/12/20 09:51:39 INFO: 2c16fb63879f0d98 became follower at term 6 raft2021/12/20 09:51:39 INFO: newRaft 2c16fb63879f0d98 [peers: [], term: 6, commit: 3, applied: 0, lastindex: 3, lastterm: 1] 2021-12-20 09:51:39.382917 W | auth: simple token is not cryptographically signed 2021-12-20 09:51:39.385788 I | etcdserver: starting server... [version: 3.4.16, cluster version: to_be_decided] raft2021/12/20 09:51:39 INFO: 2c16fb63879f0d98 switched to configuration voters=(3177002992052145560) 2021-12-20 09:51:39.387487 I | etcdserver/membership: added member 2c16fb63879f0d98 [http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 raft2021/12/20 09:51:39 INFO: 2c16fb63879f0d98 switched to configuration voters=(3177002992052145560 4607663787011114975) 2021-12-20 09:51:39.387566 I | etcdserver/membership: added member 3ff1b5cd453a87df [http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 2021-12-20 09:51:39.387580 I | rafthttp: starting peer 3ff1b5cd453a87df... 2021-12-20 09:51:39.387648 I | rafthttp: started HTTP pipelining with peer 3ff1b5cd453a87df 2021-12-20 09:51:39.388954 I | embed: listening for peers on [::]:2380 2021-12-20 09:51:39.389430 I | rafthttp: started streaming with peer 3ff1b5cd453a87df (writer) 2021-12-20 09:51:39.389659 I | rafthttp: started streaming with peer 3ff1b5cd453a87df (writer) 2021-12-20 09:51:39.390153 I | rafthttp: started peer 3ff1b5cd453a87df 2021-12-20 09:51:39.390178 I | rafthttp: added peer 3ff1b5cd453a87df raft2021/12/20 09:51:39 INFO: 2c16fb63879f0d98 switched to configuration voters=(3177002992052145560 4607663787011114975 10381479693335928327) 2021-12-20 09:51:39.390434 I | etcdserver/membership: added member 90126cc714381e07 [http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 2021-12-20 09:51:39.390449 I | rafthttp: starting peer 90126cc714381e07... 2021-12-20 09:51:39.390486 I | rafthttp: started HTTP pipelining with peer 90126cc714381e07 2021-12-20 09:51:39.393275 I | rafthttp: started streaming with peer 3ff1b5cd453a87df (stream Message reader) 2021-12-20 09:51:39.394028 I | rafthttp: started streaming with peer 3ff1b5cd453a87df (stream MsgApp v2 reader) 2021-12-20 09:51:39.394116 I | rafthttp: started streaming with peer 90126cc714381e07 (writer) 2021-12-20 09:51:39.394321 I | rafthttp: started streaming with peer 90126cc714381e07 (writer) 2021-12-20 09:51:39.394911 I | rafthttp: started peer 90126cc714381e07 2021-12-20 09:51:39.394929 I | rafthttp: added peer 90126cc714381e07 2021-12-20 09:51:39.395228 I | rafthttp: started streaming with peer 90126cc714381e07 (stream MsgApp v2 reader) 2021-12-20 09:51:39.395811 I | rafthttp: started streaming with peer 90126cc714381e07 (stream Message reader) raft2021/12/20 09:51:40 INFO: 2c16fb63879f0d98 is starting a new election at term 6 raft2021/12/20 09:51:40 INFO: 2c16fb63879f0d98 became candidate at term 7 raft2021/12/20 09:51:40 INFO: 2c16fb63879f0d98 received MsgVoteResp from 2c16fb63879f0d98 at term 7 raft2021/12/20 09:51:40 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 3ff1b5cd453a87df at term 7 raft2021/12/20 09:51:40 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 90126cc714381e07 at term 7 raft2021/12/20 09:51:42 INFO: 2c16fb63879f0d98 is starting a new election at term 7 raft2021/12/20 09:51:42 INFO: 2c16fb63879f0d98 became candidate at term 8 raft2021/12/20 09:51:42 INFO: 2c16fb63879f0d98 received MsgVoteResp from 2c16fb63879f0d98 at term 8 raft2021/12/20 09:51:42 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 3ff1b5cd453a87df at term 8 raft2021/12/20 09:51:42 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 90126cc714381e07 at term 8 raft2021/12/20 09:51:43 INFO: 2c16fb63879f0d98 is starting a new election at term 8 raft2021/12/20 09:51:43 INFO: 2c16fb63879f0d98 became candidate at term 9 raft2021/12/20 09:51:43 INFO: 2c16fb63879f0d98 received MsgVoteResp from 2c16fb63879f0d98 at term 9 raft2021/12/20 09:51:43 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 3ff1b5cd453a87df at term 9 raft2021/12/20 09:51:43 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 90126cc714381e07 at term 9 2021-12-20 09:51:44.394254 W | rafthttp: health check for peer 3ff1b5cd453a87df could not connect: dial tcp 10.244.225.135:2380: connect: connection refused 2021-12-20 09:51:44.394556 W | rafthttp: health check for peer 3ff1b5cd453a87df could not connect: dial tcp 10.244.225.135:2380: connect: connection refused 2021-12-20 09:51:44.396480 W | rafthttp: health check for peer 90126cc714381e07 could not connect: dial tcp 10.244.253.199:2380: connect: connection refused 2021-12-20 09:51:44.396529 W | rafthttp: health check for peer 90126cc714381e07 could not connect: dial tcp 10.244.253.199:2380: connect: connection refused raft2021/12/20 09:51:45 INFO: 2c16fb63879f0d98 is starting a new election at term 9 raft2021/12/20 09:51:45 INFO: 2c16fb63879f0d98 became candidate at term 10 raft2021/12/20 09:51:45 INFO: 2c16fb63879f0d98 received MsgVoteResp from 2c16fb63879f0d98 at term 10 raft2021/12/20 09:51:45 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 3ff1b5cd453a87df at term 10 raft2021/12/20 09:51:45 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 90126cc714381e07 at term 10 2021-12-20 09:51:46.387144 E | etcdserver: publish error: etcdserver: request timed out raft2021/12/20 09:51:46 INFO: 2c16fb63879f0d98 is starting a new election at term 10 raft2021/12/20 09:51:46 INFO: 2c16fb63879f0d98 became candidate at term 11 raft2021/12/20 09:51:46 INFO: 2c16fb63879f0d98 received MsgVoteResp from 2c16fb63879f0d98 at term 11 raft2021/12/20 09:51:46 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 3ff1b5cd453a87df at term 11 raft2021/12/20 09:51:46 INFO: 2c16fb63879f0d98 [logterm: 1, index: 3] sent MsgVote request to 90126cc714381e07 at term 11 2021-12-20 09:51:46.596065 C | etcdserver: failed to save state and entries error: input/output error` ### Environment - **kubernetes version** (v1.19.9) - **helm version** (version.BuildInfo{Version:"v3.7.2", GitCommit:"663a896f4a815053445eec4153677ddc24a0a361", GitTreeState:"clean", GoVersion:"go1.16.10"}) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
