tranbaokhanh opened a new issue #212: URL: https://github.com/apache/apisix-helm-chart/issues/212
### Issue description when i install APISIX via Helm with: _helm install apisix apisix/apisix --set gateway.type=NodePort --set ingress-controller.enabled=true --set etcd.persistence.size=15G --set etcd.persistence.enabled=true --set etcd.volumePermissions.enabled=true --set admin.allow.ipList="" --namespace apisix_ 1/3 node etcd running and 2/3 stuck in Crashloopbackoff with des "Containers with unready status: [etcd]". Log bellow: <html> <body> <!--StartFragment--> etcd 04:40:35.11 -- Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.11 Welcome to the Bitnami etcd container Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.11 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-etcd Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.12 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-etcd/issues Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.12 Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.12 INFO ==> ** Starting etcd setup ** Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.14 INFO ==> Validating settings in ETCD_* env vars.. Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.14 WARN ==> You set the environment variable ALLOW_NONE_AUTHENTICATION=yes. For safety reasons, do not use this flag in a production environment. Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.15 INFO ==> Initializing etcd Mon, Jan 10 2022 11:40:35 am | etcd 04:40:35.16 INFO ==> Detected data from previous deployments Mon, Jan 10 2022 11:40:50 am | etcd 04:40:50.37 WARN ==> Cluster not responding! Mon, Jan 10 2022 11:40:50 am | etcd 04:40:50.38 WARN ==> Disaster recovery is disabled, the cluster will try to recover on it's own Mon, Jan 10 2022 11:40:50 am | etcd 04:40:50.38 INFO ==> ** etcd setup finished! ** Mon, Jan 10 2022 11:40:50 am | Mon, Jan 10 2022 11:40:50 am | etcd 04:40:50.41 INFO ==> ** Starting etcd ** Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436320 I \| pkg/flags: recognized and used environment variable ETCD_ADVERTISE_CLIENT_URLS=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2379 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436563 I \| pkg/flags: recognized and used environment variable ETCD_AUTO_TLS=false Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436579 I \| pkg/flags: recognized and used environment variable ETCD_CLIENT_CERT_AUTH=false Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436592 I \| pkg/flags: recognized and used environment variable ETCD_DATA_DIR=/bitnami/etcd/data Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436638 I \| pkg/flags: recognized and used environment variable ETCD_INITIAL_ADVERTISE_PEER_URLS=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436675 I \| pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER=apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436687 I \| pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_STATE=new Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436693 I \| pkg/flags: recognized and used environment variable ETCD_INITIAL_CLUSTER_TOKEN=etcd-cluster-k8s Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436714 I \| pkg/flags: recognized and used environment variable ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436742 I \| pkg/flags: recognized and used environment variable ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436751 I \| pkg/flags: recognized and used environment variable ETCD_LOG_LEVEL=info Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436765 I \| pkg/flags: recognized and used environment variable ETCD_NAME=apisix-etcd-0 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436778 I \| pkg/flags: recognized and used environment variable ETCD_PEER_AUTO_TLS=false Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436817 W \| pkg/flags: unrecognized environment variable ETCD_TRUSTED_CA_FILE= Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436837 W \| pkg/flags: unrecognized environment variable ETCD_ON_K8S=yes Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436842 W \| pkg/flags: unrecognized environment variable ETCD_SNAPSHOTS_DIR=/snapshots Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436848 W \| pkg/flags: unrecognized environment variable ETCD_BIN_DIR=/opt/bitnami/etcd/sbin Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436852 W \| pkg/flags: unrecognized environment variable ETCD_VOLUME_DIR=/bitnami/etcd Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436859 W \| pkg/flags: unrecognized environment variable ETCD_ROOT_PASSWORD= Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436874 W \| pkg/flags: unrecognized environment variable ETCD_CLUSTER_DOMAIN=apisix-etcd-headless.apisix.svc.cluster.local Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436880 W \| pkg/flags: unrecognized environment variable ETCD_DISASTER_RECOVERY=no Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436885 W \| pkg/flags: unrecognized environment variable ETCD_KEY_FILE= Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436890 W \| pkg/flags: unrecognized environment variable ETCD_DAEMON_GROUP=etcd Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436895 W \| pkg/flags: unrecognized environment variable ETCD_START_FROM_SNAPSHOT=no Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436901 W \| pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOT_FILENAME= Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436913 W \| pkg/flags: unrecognized environment variable ETCD_INIT_SNAPSHOTS_DIR=/init-snapshot Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436922 W \| pkg/flags: unrecognized environment variable ETCD_BASE_DIR=/opt/bitnami/etcd Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436929 W \| pkg/flags: unrecognized environment variable ETCD_CERT_FILE= Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436934 W \| pkg/flags: unrecognized environment variable ETCD_NEW_MEMBERS_ENV_FILE=/bitnami/etcd/data/new_member_envs Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.436939 W \| pkg/flags: unrecognized environment variable ETCD_DAEMON_USER=etcd Mon, Jan 10 2022 11:40:50 am | [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437017 I \| etcdmain: etcd Version: 3.4.16 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437026 I \| etcdmain: Git SHA: d19fbe541 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437030 I \| etcdmain: Go Version: go1.12.17 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437034 I \| etcdmain: Go OS/Arch: linux/amd64 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437039 I \| etcdmain: setting maximum number of CPUs to 8, total number of available CPUs is 8 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437153 N \| etcdmain: the server is already initialized as member before, starting as etcd member... Mon, Jan 10 2022 11:40:50 am | [WARNING] Deprecated '--logger=capnslog' flag is set; use '--logger=zap' flag instead Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437598 I \| embed: name = apisix-etcd-0 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437670 I \| embed: data dir = /bitnami/etcd/data Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437677 I \| embed: member dir = /bitnami/etcd/data/member Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437682 I \| embed: heartbeat = 100ms Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437687 I \| embed: election = 1000ms Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437691 I \| embed: snapshot count = 100000 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437703 I \| embed: advertise client URLs = http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2379 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437711 I \| embed: initial advertise peer URLs = http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.437718 I \| embed: initial cluster = Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.446933 I \| etcdserver: restarting member 90126cc714381e07 in cluster b0d7015fda1525c8 at commit index 3 Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: 90126cc714381e07 switched to configuration voters=() Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: 90126cc714381e07 became follower at term 1372 Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: newRaft 90126cc714381e07 [peers: [], term: 1372, commit: 3, applied: 0, lastindex: 3, lastterm: 1] Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.451256 W \| auth: simple token is not cryptographically signed Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.468320 I \| etcdserver: starting server... [version: 3.4.16, cluster version: to_be_decided] Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: 90126cc714381e07 switched to configuration voters=(3177002992052145560) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.470597 I \| etcdserver/membership: added member 2c16fb63879f0d98 [http://apisix-etcd-1.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.470654 I \| rafthttp: starting peer 2c16fb63879f0d98... Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.470703 I \| rafthttp: started HTTP pipelining with peer 2c16fb63879f0d98 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.475032 I \| embed: listening for peers on [::]:2380 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478696 I \| rafthttp: started streaming with peer 2c16fb63879f0d98 (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478757 I \| rafthttp: started streaming with peer 2c16fb63879f0d98 (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478800 I \| rafthttp: started peer 2c16fb63879f0d98 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478827 I \| rafthttp: added peer 2c16fb63879f0d98 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478869 I \| rafthttp: started streaming with peer 2c16fb63879f0d98 (stream MsgApp v2 reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.478974 I \| rafthttp: started streaming with peer 2c16fb63879f0d98 (stream Message reader) Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: 90126cc714381e07 switched to configuration voters=(3177002992052145560 4607663787011114975) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.479095 I \| etcdserver/membership: added member 3ff1b5cd453a87df [http://apisix-etcd-2.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.479134 I \| rafthttp: starting peer 3ff1b5cd453a87df... Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.479192 I \| rafthttp: started HTTP pipelining with peer 3ff1b5cd453a87df Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.481664 I \| rafthttp: started streaming with peer 3ff1b5cd453a87df (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.484246 I \| rafthttp: started streaming with peer 3ff1b5cd453a87df (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.488643 I \| rafthttp: started peer 3ff1b5cd453a87df Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.488699 I \| rafthttp: added peer 3ff1b5cd453a87df Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.488725 I \| rafthttp: started streaming with peer 3ff1b5cd453a87df (stream MsgApp v2 reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.488748 I \| rafthttp: started streaming with peer 3ff1b5cd453a87df (stream Message reader) Mon, Jan 10 2022 11:40:50 am | raft2022/01/10 04:40:50 INFO: 90126cc714381e07 switched to configuration voters=(3177002992052145560 4607663787011114975 10381479693335928327) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.489007 I \| etcdserver/membership: added member 90126cc714381e07 [http://apisix-etcd-0.apisix-etcd-headless.apisix.svc.cluster.local:2380] to cluster b0d7015fda1525c8 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492210 E \| etcdserver: the member has been permanently removed from the cluster Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492240 I \| etcdserver: the data-dir used by this member must be removed. Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492301 E \| etcdserver: publish error: etcdserver: request cancelled Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492329 I \| etcdserver: aborting publish because server is stopped Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492356 I \| rafthttp: stopping peer 2c16fb63879f0d98... Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492375 I \| rafthttp: stopped streaming with peer 2c16fb63879f0d98 (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492387 I \| rafthttp: stopped streaming with peer 2c16fb63879f0d98 (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492414 I \| rafthttp: stopped HTTP pipelining with peer 2c16fb63879f0d98 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492433 I \| rafthttp: stopped streaming with peer 2c16fb63879f0d98 (stream MsgApp v2 reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492446 I \| rafthttp: stopped streaming with peer 2c16fb63879f0d98 (stream Message reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492454 I \| rafthttp: stopped peer 2c16fb63879f0d98 Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492460 I \| rafthttp: stopping peer 3ff1b5cd453a87df... Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492469 I \| rafthttp: stopped streaming with peer 3ff1b5cd453a87df (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492478 I \| rafthttp: stopped streaming with peer 3ff1b5cd453a87df (writer) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492496 I \| rafthttp: stopped HTTP pipelining with peer 3ff1b5cd453a87df Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492579 I \| rafthttp: stopped streaming with peer 3ff1b5cd453a87df (stream MsgApp v2 reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492599 I \| rafthttp: stopped streaming with peer 3ff1b5cd453a87df (stream Message reader) Mon, Jan 10 2022 11:40:50 am | 2022-01-10 04:40:50.492607 I \| rafthttp: stopped peer 3ff1b5cd453a87df <!--EndFragment--> </body> </html> ### Environment - apisix version (cmd: `apisix version`): apache/apisix:2.10.0-alpine - OS (cmd: `uname -a`):Linux node01 4.15.0-118-generic apache/apisix#119-Ubuntu SMP Tue Sep 8 12:30:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux - OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`): - etcd version, if have (cmd: run `curl http://127.0.0.1:9090/v1/server_info` to get the info from server-info API): - apisix-dashboard version, if have: - the plugin runner version, if the issue is about a plugin runner (cmd: depended on the kind of runner): - luarocks version, if the issue is about installation (cmd: `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
