[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-833164013 > Any news? We found the similar porblem, some of nodes (not all) found this error message, my etcd version is 3.4.13. @nanamikon will add PR to solve it this week -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-787712261 > I'm not sure whether you set these in etcd's Statefulset templates fields. Hi @tokers it seems @GBXing deploy etcd with docker but not k8s, so that might not be the solution This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-787711777 @GBXing Got it, the error log is the same. Could you also show your `config.yaml`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-787710873 > v3.lua:593: attempt to index field 'result' (a nil value) is this a bug of apisix？ Yes I do think so, we missed some test coverage, so no related error messages are prepared for this This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673:
URL: https://github.com/apache/apisix/issues/3673#issuecomment-787094518
Hi @GBXing, I tried to reproduce the problem, and the result seems a bit
different from what you get. My reproduce step is:
1. Configure etcd host with domain name:
```
DNS_IP=$(kubectl get svc -n kube-system -l k8s-app=kube-dns -o
'jsonpath={..spec.clusterIP}')
echo "dns_resolver:
- ${DNS_IP}
etcd:
host:
- \\"\\"
" > ./conf/config.yaml
```
2. Setup APISIX and everything works as expect
3. Kill leader/follower pod of etcd ( gives me the same result ), and the
error log would produce:
```
# Multiple of
2021/02/27 15:30:19 [error] 49#49: *114289 [lua] config_etcd.lua:566:
failed to fetch data from etcd:
/usr/local/apisix//deps/share/lua/5.1/resty/etcd/v3.lua:593: attempt to index
field 'result' (a nil value)
stack traceback:
/usr/local/apisix//deps/share/lua/5.1/resty/etcd/v3.lua:593: in function
'res_func'
/usr/local/apisix/apisix/core/config_etcd.lua:125: in function 'waitdir'
/usr/local/apisix/apisix/core/config_etcd.lua:305: in function
'sync_data'
/usr/local/apisix/apisix/core/config_etcd.lua:540: in function
[C]: in function 'xpcall'
/usr/local/apisix/apisix/core/config_etcd.lua:530: in function
, etcd key:
/apisix/global_rules, context: ngx.timer
# Multiple of
2021/02/27 15:30:38 [error] 53#53: *113602 [lua] config_etcd.lua:544:
failed to fetch data from etcd: connection refused, etcd key: /apisix/ssl,
context: ngx.timer
```
With etcd-operator, etcd got unreachable for seconds and returned back
to normal.
Is there any places I missed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-785885676 @GBXing You could give it a try on master branch (`apisix:dev` for docker tag) after #3676 got merged if it's urgent. Or if it's more urgent you could try to [build docker image from local code](https://github.com/apache/apisix-docker#build-an-image-from-source) to have a test. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [apisix] Yiyiyimu commented on issue #3673: request help: A node in the K8S environment ETCD cluster died, causing Apisix to fail
Yiyiyimu commented on issue #3673: URL: https://github.com/apache/apisix/issues/3673#issuecomment-785863392 > will chaos mesh cover this？ I'll make a test tomorrow This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
