shayktrust opened a new issue, #768:
URL: https://github.com/apache/apisix-helm-chart/issues/768
### Name and Version
targetRevision: v2.7.0
###EKS Version
1.29
### What architecture are you using?
amd64
### What steps will reproduce the bug?
Deploy the chart
### Are you using any custom parameters or values?
Yes
### What is the expected behavior?
apisix-etcd-0 apisix-etcd-1 and apisix-etcd-2 in a Running mode
### What do you see instead?
CrashLoopBackOff
```
{"level":"warn","ts":"2024-07-08T10:17:39.664Z","caller":"etcdserver/server.go:1127","msg":"server
error","error":"the member has been permanently removed from the cluster"}
{"level":"warn","ts":"2024-07-08T10:17:39.664Z","caller":"etcdserver/server.go:1128","msg":"data-dir
used by this member must be removed"}
{"level":"warn","ts":"2024-07-08T10:17:39.665Z","caller":"etcdserver/server.go:2083","msg":"failed
to publish local member to cluster through
raft","local-member-id":"dca459b91c9da974","local-member-attributes":"{Name:apisix-etcd-0
ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.api-gw.svc.cluster.local:2379
http://apisix-etcd.api-gw.svc.cluster.local:2379]}","request-path":"/0/members/dca459b91c9da974/attributes","publish-timeout":"7s","error":"etcdserver:
request cancelled"}
{"level":"warn","ts":"2024-07-08T10:17:39.665Z","caller":"etcdserver/server.go:2083","msg":"failed
to publish local member to cluster through
raft","local-member-id":"dca459b91c9da974","local-member-attributes":"{Name:apisix-etcd-0
ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.api-gw.svc.cluster.local:2379
http://apisix-etcd.api-gw.svc.cluster.local:2379]}","request-path":"/0/members/dca459b91c9da974/attributes","publish-timeout":"7s","error":"etcdserver:
request cancelled"}
{"level":"warn","ts":"2024-07-08T10:17:39.665Z","caller":"etcdserver/server.go:2083","msg":"failed
to publish local member to cluster through
raft","local-member-id":"dca459b91c9da974","local-member-attributes":"{Name:apisix-etcd-0
ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.api-gw.svc.cluster.local:2379
http://apisix-etcd.api-gw.svc.cluster.local:2379]}","request-path":"/0/members/dca459b91c9da974/attributes","publish-timeout":"7s","error":"etcdserver:
request cancelled"}
{"level":"warn","ts":"2024-07-08T10:17:39.665Z","caller":"etcdserver/server.go:2073","msg":"stopped
publish because server is
stopped","local-member-id":"dca459b91c9da974","local-member-attributes":"{Name:apisix-etcd-0
ClientURLs:[http://apisix-etcd-0.apisix-etcd-headless.api-gw.svc.cluster.local:2379
http://apisix-etcd.api-gw.svc.cluster.local:2379]}","publish-timeout":"7s","error":"etcdserver:
server stopped"}
```
```
kubectl get all -n api-gw
NAME READY STATUS
RESTARTS AGE
pod/apisix-6fdf6b9c66-64z2f 1/1 Running
0 17h
pod/apisix-6fdf6b9c66-brsjk 1/1 Running
0 17h
pod/apisix-6fdf6b9c66-jz2rk 1/1 Running
0 17h
pod/apisix-etcd-0 0/1 CrashLoopBackOff
203 (38s ago) 17h
pod/apisix-etcd-1 0/1 CrashLoopBackOff
202 (5m2s ago) 17h
pod/apisix-etcd-2 1/1 Running
0 17h
pod/apisix-ingress-controller-844c65bfdf-5v799 1/1 Running
0 17h
pod/apisix-ingress-controller-844c65bfdf-fzhvg 1/1 Running
0 22h
NAME TYPE CLUSTER-IP
EXTERNAL-IP PORT(S) AGE
service/apisix-admin ClusterIP 172.20.187.18
<none> 9180/TCP 22h
service/apisix-etcd ClusterIP
172.20.177.186 <none> 2379/TCP,2380/TCP 22h
service/apisix-etcd-headless ClusterIP None
<none> 2379/TCP,2380/TCP 22h
service/apisix-gateway NodePort 172.20.111.3
<none> 80:31196/TCP,443:31359/TCP 22h
service/apisix-ingress-controller ClusterIP
172.20.246.190 <none> 80/TCP 22h
service/apisix-ingress-controller-apisix-gateway NodePort
172.20.219.206 <none> 80:30461/TCP,443:31371/TCP 22h
NAME READY UP-TO-DATE AVAILABLE
AGE
deployment.apps/apisix 3/3 3 3
22h
deployment.apps/apisix-ingress-controller 2/2 2 2
22h
NAME DESIRED CURRENT
READY AGE
replicaset.apps/apisix-6fdf6b9c66 3 3 3
22h
replicaset.apps/apisix-ingress-controller-844c65bfdf 2 2 2
22h
NAME READY AGE
statefulset.apps/apisix-etcd 1/3 22h
NAME REFERENCE TARGETS
MINPODS MAXPODS REPLICAS AGE
horizontalpodautoscaler.autoscaling/apisix Deployment/apisix 7%/80%,
61%/80% 3 6 3 22h
```
```
kubectl describe pods -n api-gw apisix-etcd-0
Name: apisix-etcd-0
Namespace: api-gw
Priority: 0
Service Account: default
Node: ip-10-0-18-102.eu-north-1.compute.internal/10.0.18.102
Start Time: Sun, 07 Jul 2024 20:15:28 +0300
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
apps.kubernetes.io/pod-index=0
controller-revision-hash=apisix-etcd-5d9864fd68
helm.sh/chart=etcd-8.7.7
statefulset.kubernetes.io/pod-name=apisix-etcd-0
Annotations: checksum/token-secret:
622d20823882c1300c1be66970c8a4304a57e6d674f4c7da8a29e8e8062bb7c1
Status: Running
IP: 10.0.18.31
IPs:
IP: 10.0.18.31
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID:
containerd://3e7d388fe249ab387b0f2af890addeffc3fe592b8ee8f4e47362d2e6dd33f13a
Image: docker.io/bitnami/etcd:3.5.7-debian-11-r14
Image ID:
docker.io/bitnami/etcd@sha256:0825cafa1c5f0c97d86009f3af8c0f5a9d4279fcfdeb0a2a09b84a1eb7893a13
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 08 Jul 2024 13:27:59 +0300
Finished: Mon, 08 Jul 2024 13:28:04 +0300
Ready: False
Restart Count: 203
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-0
(v1:metadata.name)
MY_STS_NAME: apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN:
jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.api-gw.svc.cluster.local:2379,http://apisix-etcd.api-gw.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS:
http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.api-gw.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS:
http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: new
ETCD_INITIAL_CLUSTER:
apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.api-gw.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.api-gw.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.api-gw.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN:
apisix-etcd-headless.api-gw.svc.cluster.local
NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME: dev
NEW_RELIC_METADATA_KUBERNETES_NODE_NAME:
(v1:spec.nodeName)
NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME: api-gw
(v1:metadata.namespace)
NEW_RELIC_METADATA_KUBERNETES_POD_NAME: apisix-etcd-0
(v1:metadata.name)
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME: etcd
NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME:
docker.io/bitnami/etcd:3.5.7-debian-11-r14
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from
kube-api-access-h8wk9 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-0
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: apisix-etcd-jwt-token
Optional: false
kube-api-access-h8wk9:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute
op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning BackOff 2s (x4811 over 17h) kubelet Back-off restarting failed
container etcd in pod apisix-etcd-0_api-gw(607771a9-2674-40b5-a6d5-4be0971f0706)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
