hansedong commented on issue #9336:
URL: https://github.com/apache/apisix/issues/9336#issuecomment-1534194681
@kingluo
I built APISIX based on the latest code. When I use gRPC to connect to Etcd,
I encounter the following error message:
```
2023/05/04 14:23:50 [warn] 500026#500026: *4 [lua] plugin.lua:202: load():
new plugins:
{"udp-logger":true,"client-control":true,"cors":true,"batch-requests":true,"gzip":true,"syslog":true,"authz-keycloak":true,"zipkin":true,"forward-auth":true,"tcp-logger":true,"aws-lambda":true,"skywalking-logger":true,"fault-injection":true,"redirect":true,"http-logger":true,"cas-auth":true,"kafka-logger":true,"openid-connect":true,"kafka-proxy":true,"authz-casdoor":true,"loggly":true,"uri-blocker":true,"key-auth":true,"azure-functions":true,"limit-conn":true,"basic-auth":true,"openfunction":true,"public-api":true,"datadog":true,"clickhouse-logger":true,"opa":true,"proxy-control":true,"proxy-cache":true,"csrf":true,"elasticsearch-logger":true,"limit-req":true,"ext-plugin-post-resp":true,"ext-plugin-post-req":true,"server-info":true,"jwt-auth":true,"openwhisk":true,"degraphql":true,"traffic-split":true,"log-rotate":true,"google-cloud-logging":true,"wolf-rbac":true,"inspect":true,"api-breaker":tr
ue,"real-ip":true,"echo":true,"rocketmq-logger":true,"file-logger":true,"node-status":true,"mocking":true,"body-transformer":true,"sls-logger":true,"splunk-hec-logging":true,"consumer-restriction":true,"ext-plugin-pre-req":true,"referer-restriction":true,"ua-restriction":true,"workflow":true,"authz-casbin":true,"ip-restriction":true,"request-validation":true,"hmac-auth":true,"grpc-web":true,"response-rewrite":true,"example-plugin":true,"limit-count":true,"proxy-rewrite":true,"serverless-pre-function":true,"prometheus":true,"request-id":true,"grpc-transcode":true,"proxy-mirror":true,"ldap-auth":true,"tencent-cloud-cls":true,"serverless-post-function":true,"ai":true},
context: init_worker_by_lua*
2023/05/04 14:23:50 [warn] 500025#500025: *2 [lua] plugin.lua:252:
load_stream(): new plugins:
{"prometheus":true,"syslog":true,"mqtt-proxy":true,"limit-conn":true,"ip-restriction":true},
context: init_worker_by_lua*
2023/05/04 14:23:50 [warn] 500027#500027: *3 [lua] plugin.lua:252:
load_stream(): new plugins:
{"prometheus":true,"syslog":true,"mqtt-proxy":true,"limit-conn":true,"ip-restriction":true},
context: init_worker_by_lua*
2023/05/04 14:23:50 [warn] 500026#500026: *4 [lua] plugin.lua:252:
load_stream(): new plugins:
{"prometheus":true,"syslog":true,"mqtt-proxy":true,"limit-conn":true,"ip-restriction":true},
context: init_worker_by_lua*
2023/05/04 14:23:50 [warn] 500024#500024: *5 [lua] plugin.lua:252:
load_stream(): new plugins:
{"prometheus":true,"syslog":true,"mqtt-proxy":true,"limit-conn":true,"ip-restriction":true},
context: init_worker_by_lua*
2023/05/04 14:23:50 [warn] 500031#500031: *1 [lua] plugin.lua:252:
load_stream(): new plugins:
{"prometheus":true,"syslog":true,"mqtt-proxy":true,"limit-conn":true,"ip-restriction":true},
context: init_worker_by_lua*
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.67:2379",
host: "localhost"
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.68:2379",
host: "localhost"
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.67:2379",
host: "localhost"
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.66:2379",
host: "localhost"
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.66:2379",
host: "localhost"
2023/05/04 14:24:51 [error] 500025#500025: *19 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /etcdserverpb.Watch/Watch HTTP/2.0", upstream: "grpcs://10.72.8.68:2379",
host: "localhost"
```
It can be clearly seen that the time elapsed from starting APISIX to
encountering the following error is about 1 minute, Exactly speaking, it's 61
seconds!!
Here is part of my configuration of APISIX about Etcd:
```
etcd:
host: # it's possible to define multiple etcd
hosts addresses of the same etcd cluster.
- "https://10.72.8.66:2379"; # multiple etcd address, if your etcd
cluster enables TLS, please use https scheme,
- "https://10.72.8.67:2379"; # e.g. https://127.0.0.1:2379.
- "https://10.72.8.68:2379";
prefix: /apisix # configuration prefix in etcd
use_grpc: true # enable the experimental configuration
sync via gRPC
timeout: 300 # 30 seconds. Use a much higher timeout
(like an hour) if the `use_grpc` is true.
#resync_delay: 5 # when sync failed and a rest is needed,
resync after the configured seconds plus 50% random jitter
#health_check_timeout: 10 # etcd retry the unhealthy nodes after
the configured seconds
startup_retry: 2 # the number of retry to etcd during the
startup, default to 2
#user: root # root username for etcd
#password: 5tHkHhYkjr6cQY # root password for etcd
```
What needs to be emphasized is that after I changed the timeout
configuration item to 30, the above error no longer appeared (observed for
several hours without appearing), and as long as the value of timeout exceeds
60, the above error will reappear after 1 minute( Exactly 61 seconds ).
Above, I mentioned using gRPC to connect to Etcd and error occurs when the
timeout value exceeds 60. If I switch APISIX's connection to Etcd from gRPC
back to HTTP (use_grpc: false), there are still issues. Below is the specific
error:
```
2023/05/04 14:58:42 [error] 503639#503639: *69 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.67:2379/v3/watch";, host:
"127.0.0.1"
2023/05/04 14:58:42 [error] 503639#503639: *73 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.68:2379/v3/watch";, host:
"127.0.0.1"
2023/05/04 14:58:42 [error] 503639#503639: *79 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.67:2379/v3/watch";, host:
"127.0.0.1"
2023/05/04 14:58:42 [error] 503645#503645: *6 [lua] config_etcd.lua:634:
failed to fetch data from etcd: closed, etcd key: /apisix/plugin_metadata,
context: ngx.timer
2023/05/04 14:58:42 [error] 503645#503645: *10 [lua] config_etcd.lua:634:
failed to fetch data from etcd: closed, etcd key: /apisix/ssl, context:
ngx.timer
2023/05/04 14:58:42 [error] 503639#503639: *77 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.67:2379/v3/watch";, host:
"127.0.0.1"
2023/05/04 14:58:42 [error] 503645#503645: *16 [lua] config_etcd.lua:634:
failed to fetch data from etcd: closed, etcd key: /apisix/plugin_configs,
context: ngx.timer
2023/05/04 14:58:42 [error] 503639#503639: *37 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.66:2379/v3/watch";, host:
"127.0.0.1"
2023/05/04 14:58:42 [error] 503645#503645: *14 [lua] config_etcd.lua:634:
failed to fetch data from etcd: closed, etcd key: /apisix/services, context:
ngx.timer
2023/05/04 14:58:42 [error] 503645#503645: *4 [lua] config_etcd.lua:634:
failed to fetch data from etcd: closed, etcd key: /apisix/protos, context:
ngx.timer
2023/05/04 14:58:42 [error] 503639#503639: *81 upstream timed out (110:
Connection timed out) while reading upstream, client: unix:, server: , request:
"POST /v3/watch HTTP/1.1", upstream: "https://10.72.8.68:2379/v3/watch";, host:
"127.0.0.1"
```
Similarly, after testing, no errors will occur when the value of the
'timeout' parameter is below 60.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
