jishaashokan commented on issue #156:
URL:
https://github.com/apache/apisix-helm-chart/issues/156#issuecomment-1375089394
Have the same issue with the etcd crashing in my EKS cluster. Have tried
deleting the PVC and reinstalling apisix via helm multiple times, however, etcd
keeps crashing.
`mk describe pod/apisix-etcd-2
Name: apisix-etcd-2
Namespace: ingress-apisix
Priority: 0
Node: ip-172-31-109-173.ap-south-1.compute.internal/172.31.109.173
Start Time: Mon, 09 Jan 2023 09:14:01 +0530
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=apisix-etcd-54d7f4b448
helm.sh/chart=etcd-8.3.4
statefulset.kubernetes.io/pod-name=apisix-etcd-2
Annotations: checksum/token-secret:
7f28bd39a9649b0425cff5da399d496cfadede8bb85303b4edf589da3bc2e751
kubernetes.io/psp: eks.privileged
Status: Running
IP: 172.31.111.200
IPs:
IP: 172.31.111.200
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID:
docker://35e9a0fa4f62390a84a04e7c1a3761af83595a78e3dfdf9b008f944ce044d152
Image: docker.io/bitnami/etcd:3.5.4-debian-11-r14
Image ID:
docker-pullable://bitnami/etcd@sha256:e5ef30fa508e6f3a028a4e26acc7ec2803eea1370dc9c1da692ee0405cdaf50d
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 09 Jan 2023 09:14:07 +0530
Ready: True
Restart Count: 0
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-2 (v1:metadata.name)
MY_STS_NAME: apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN:
jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: existing
ETCD_INITIAL_CLUSTER:
apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN:
apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from
kube-api-access-9hfvq (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-2
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: apisix-etcd-jwt-token
Optional: false
kube-api-access-9hfvq:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute
op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 31m default-scheduler Successfully assigned
ingress-apisix/apisix-etcd-2 to ip-172-31-109-173.ap-south-1.compute.internal
Normal Pulled 31m kubelet Container image
"docker.io/bitnami/etcd:3.5.4-debian-11-r14" already present on machine
Normal Created 31m kubelet Created container etcd
Normal Started 31m kubelet Started container etcd
jisha@jisha-Latitude-3420:~/Projects/gRPC/grpc-strest$
jisha@jisha-Latitude-3420:~/Projects/gRPC/grpc-strest$ mk describe
pod/apisix-etcd-1
Name: apisix-etcd-1
Namespace: ingress-apisix
Priority: 0
Node: ip-172-31-102-32.ap-south-1.compute.internal/172.31.102.32
Start Time: Mon, 09 Jan 2023 09:15:50 +0530
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=apisix-etcd-54d7f4b448
helm.sh/chart=etcd-8.3.4
statefulset.kubernetes.io/pod-name=apisix-etcd-1
Annotations: checksum/token-secret:
7f28bd39a9649b0425cff5da399d496cfadede8bb85303b4edf589da3bc2e751
kubernetes.io/psp: eks.privileged
Status: Running
IP: 172.31.100.113
IPs:
IP: 172.31.100.113
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID:
docker://759dba6c28cbcba2e077201656aede69868b4562dfc44398091c3c47a6a6a47f
Image: docker.io/bitnami/etcd:3.5.4-debian-11-r14
Image ID:
docker-pullable://bitnami/etcd@sha256:e5ef30fa508e6f3a028a4e26acc7ec2803eea1370dc9c1da692ee0405cdaf50d
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 09 Jan 2023 09:43:55 +0530
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Mon, 09 Jan 2023 09:39:55 +0530
Finished: Mon, 09 Jan 2023 09:43:55 +0530
Ready: False
Restart Count: 7
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-1 (v1:metadata.name)
MY_STS_NAME: apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN:
jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: existing
ETCD_INITIAL_CLUSTER:
apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN:
apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from
kube-api-access-vcq7m (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-1
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: apisix-etcd-jwt-token
Optional: false
kube-api-access-vcq7m:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute
op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal Scheduled 30m default-scheduler
Successfully assigned ingress-apisix/apisix-etcd-1 to
ip-172-31-102-32.ap-south-1.compute.internal
Normal SuccessfulAttachVolume 30m
attachdetach-controller AttachVolume.Attach succeeded for volume
"pvc-cd13081f-9732-4e12-b8dc-a0d7c8fad6dd"
Normal Pulled 30m kubelet
Container image "docker.io/bitnami/etcd:3.5.4-debian-11-r14" already present
on machine
Normal Created 30m kubelet
Created container etcd
Normal Started 30m kubelet
Started container etcd
Warning Unhealthy 27m (x5 over 29m) kubelet
Liveness probe failed:
Normal Killing 27m kubelet
Container etcd failed liveness probe, will be restarted
Warning FailedPreStopHook 27m kubelet
Exec lifecycle hook ([/opt/bitnami/scripts/etcd/prestop.sh]) for Container
"etcd" in Pod
"apisix-etcd-1_ingress-apisix(724e8425-d8ed-4634-81fc-c4a88b1c67da)" failed -
error: command '/opt/bitnami/scripts/etcd/prestop.sh' exited with 128: Error:
bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID
in Hex
, message: "Error: bad member ID arg (strconv.ParseUint: parsing \"\":
invalid syntax), expecting ID in Hex\n"
Warning Unhealthy 31s (x133 over 29m) kubelet Readiness probe failed:
jisha@jisha-Latitude-3420:~/Projects/gRPC/grpc-strest$ mk describe
pod/apisix-etcd-0
Name: apisix-etcd-0
Namespace: ingress-apisix
Priority: 0
Node: ip-172-31-110-110.ap-south-1.compute.internal/172.31.110.110
Start Time: Mon, 09 Jan 2023 09:12:12 +0530
Labels: app.kubernetes.io/instance=apisix
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=etcd
controller-revision-hash=apisix-etcd-5c44c455f7
helm.sh/chart=etcd-8.3.4
statefulset.kubernetes.io/pod-name=apisix-etcd-0
Annotations: checksum/token-secret:
b04b8e4c76df91b28aa60b5607a9ced3974beacab02358b3c9278e707a332628
kubernetes.io/psp: eks.privileged
Status: Running
IP: 172.31.110.252
IPs:
IP: 172.31.110.252
Controlled By: StatefulSet/apisix-etcd
Containers:
etcd:
Container ID:
docker://13366b4b07151a564ebb77bb52a05993b3f8a197d16a583a73c573233fb45c1d
Image: docker.io/bitnami/etcd:3.5.4-debian-11-r14
Image ID:
docker-pullable://bitnami/etcd@sha256:e5ef30fa508e6f3a028a4e26acc7ec2803eea1370dc9c1da692ee0405cdaf50d
Ports: 2379/TCP, 2380/TCP
Host Ports: 0/TCP, 0/TCP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 137
Started: Mon, 09 Jan 2023 09:40:17 +0530
Finished: Mon, 09 Jan 2023 09:44:17 +0530
Ready: False
Restart Count: 7
Liveness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=30s #success=1 #failure=5
Readiness: exec [/opt/bitnami/scripts/etcd/healthcheck.sh]
delay=60s timeout=5s period=10s #success=1 #failure=5
Environment:
BITNAMI_DEBUG: false
MY_POD_IP: (v1:status.podIP)
MY_POD_NAME: apisix-etcd-0 (v1:metadata.name)
MY_STS_NAME: apisix-etcd
ETCDCTL_API: 3
ETCD_ON_K8S: yes
ETCD_START_FROM_SNAPSHOT: no
ETCD_DISASTER_RECOVERY: no
ETCD_NAME: $(MY_POD_NAME)
ETCD_DATA_DIR: /bitnami/etcd/data
ETCD_LOG_LEVEL: info
ALLOW_NONE_AUTHENTICATION: yes
ETCD_AUTH_TOKEN:
jwt,priv-key=/opt/bitnami/etcd/certs/token/jwt-token.pem,sign-method=RS256,ttl=10m
ETCD_ADVERTISE_CLIENT_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2379,http://apisix-etcd.ingress-apisix.svc.cluster.local:2379
ETCD_LISTEN_CLIENT_URLS: http://0.0.0.0:2379
ETCD_INITIAL_ADVERTISE_PEER_URLS:
http://$(MY_POD_NAME).apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_LISTEN_PEER_URLS: http://0.0.0.0:2380
ETCD_INITIAL_CLUSTER_TOKEN: etcd-cluster-k8s
ETCD_INITIAL_CLUSTER_STATE: existing
ETCD_INITIAL_CLUSTER:
apisix-etcd-0=http://apisix-etcd-0.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-1=http://apisix-etcd-1.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380,apisix-etcd-2=http://apisix-etcd-2.apisix-etcd-headless.ingress-apisix.svc.cluster.local:2380
ETCD_CLUSTER_DOMAIN:
apisix-etcd-headless.ingress-apisix.svc.cluster.local
Mounts:
/bitnami/etcd from data (rw)
/opt/bitnami/etcd/certs/token/ from etcd-jwt-token (ro)
/var/run/secrets/kubernetes.io/serviceaccount from
kube-api-access-cwxz2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
data:
Type: PersistentVolumeClaim (a reference to a
PersistentVolumeClaim in the same namespace)
ClaimName: data-apisix-etcd-0
ReadOnly: false
etcd-jwt-token:
Type: Secret (a volume populated by a Secret)
SecretName: apisix-etcd-jwt-token
Optional: false
kube-api-access-cwxz2:
Type: Projected (a volume that contains injected data
from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: BestEffort
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute
op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute
op=Exists for 300s
Events:
Type Reason Age From
Message
---- ------ ---- ----
-------
Normal Scheduled 34m default-scheduler
Successfully assigned ingress-apisix/apisix-etcd-0 to
ip-172-31-110-110.ap-south-1.compute.internal
Normal SuccessfulAttachVolume 34m
attachdetach-controller AttachVolume.Attach succeeded for volume
"pvc-09001332-85bf-405a-a4e2-8579d00d9a5b"
Normal Pulled 34m kubelet
Container image "docker.io/bitnami/etcd:3.5.4-debian-11-r14" already present
on machine
Normal Created 34m kubelet
Created container etcd
Normal Started 34m kubelet
Started container etcd
Warning Unhealthy 30m (x5 over 32m) kubelet
Liveness probe failed:
Normal Killing 30m kubelet
Container etcd failed liveness probe, will be restarted
Warning FailedPreStopHook 30m kubelet
Exec lifecycle hook ([/opt/bitnami/scripts/etcd/prestop.sh]) for Container
"etcd" in Pod
"apisix-etcd-0_ingress-apisix(237e1554-1e0c-438b-be98-761988ee70f0)" failed -
error: command '/opt/bitnami/scripts/etcd/prestop.sh' exited with 128: Error:
bad member ID arg (strconv.ParseUint: parsing "": invalid syntax), expecting ID
in Hex
, message: "Error: bad member ID arg (strconv.ParseUint: parsing \"\":
invalid syntax), expecting ID in Hex\n"
Warning Unhealthy 4m16s (x133 over 33m) kubelet Readiness probe failed:
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
