ryan4yin opened a new issue #4685:
URL: https://github.com/apache/apisix/issues/4685
### Issue description
when I set the upstreams's `retries=0` to disable the retry mechanism, the
access_log shows that apisix retries many times.
access_log by example:
```
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET",
"uri": "/xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time":
"5.170", "body_bytes_sent": "154", "scheme": "http", "upstream_addr":
"10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502",
"upstream_response_time": "3.024, 2.148", "upstream_connect_time": "0.000,
1.124", "upstream_header_time": "-, -"}
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET",
"uri": "xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time":
"8.223", "body_bytes_sent": "154", "scheme": "http", "upstream_addr":
"10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502",
"upstream_response_time": "6.076, 2.148", "upstream_connect_time": "0.000,
1.124", "upstream_header_time": "-, -"}
{ "time_iso8601": "2021-07-26T20:33:30+00:00", "request_method": "GET",
"uri": "/xxx", "server_protocol": "HTTP/1.1", "status": "502", "request_time":
"2.596", "body_bytes_sent": "154", "scheme": "http", "upstream_addr":
"10.100.156.166:8080, 10.100.156.166:8080", "upstream_status": "502, 502",
"upstream_response_time": "0.444, 2.152", "upstream_connect_time": "0.000,
1.128", "upstream_header_time": "-, -"}
```
### Environment
use official docker image: `apache/apisix:2.5-alpine`
* apisix version (cmd: `apisix version`): 2.5
* OS (cmd: `uname -a`): Linux apisix-6f65f5bc9-km66f
4.14.209-160.335.amzn2.x86_64 #1 SMP Wed Dec 2 23:31:46 UTC 2020 x86_64 Linux
* OpenResty / Nginx version (cmd: `nginx -V` or `openresty -V`):
```
nginx version: openresty/1.19.3.1
built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
built with OpenSSL 1.1.1k 25 Mar 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx --with-cc-opt='-O2
-DNGX_LUA_ABORT_AT_PANIC -I/usr/local/openresty/pcre/include
-I/usr/local/openresty/openssl/include' --add-module=../ngx_devel_kit-0.3.1
--add-module=../echo-nginx-module-0.62 --add-module=../xss-nginx-module-0.06
--add-module=../ngx_coolkit-0.2 --add-module=../set-misc-nginx-module-0.32
--add-module=../form-input-nginx-module-0.12
--add-module=../encrypted-session-nginx-module-0.08
--add-module=../srcache-nginx-module-0.32 --add-module=../ngx_lua-0.10.19
--add-module=../ngx_lua_upstream-0.07
--add-module=../headers-more-nginx-module-0.33
--add-module=../array-var-nginx-module-0.05
--add-module=../memc-nginx-module-0.19 --add-module=../redis2-nginx-module-0.15
--add-module=../redis-nginx-module-0.3.7
--add-module=../rds-json-nginx-module-0.15
--add-module=../rds-csv-nginx-module-0.09 --add-module=../ngx_stream_lua-0.0.9
--with-ld-opt='-Wl,-rpath,/usr/local/openresty/luajit/lib
-L/usr/local/openresty/pcre/l
ib -L/usr/local/openresty/openssl/lib
-Wl,-rpath,/usr/local/openresty/pcre/lib:/usr/local/openresty/openssl/lib'
--with-pcre --with-compat --with-file-aio --with-http_addition_module
--with-http_auth_request_module --with-http_dav_module --with-http_flv_module
--with-http_geoip_module=dynamic --with-http_gunzip_module
--with-http_gzip_static_module --with-http_image_filter_module=dynamic
--with-http_mp4_module --with-http_random_index_module
--with-http_realip_module --with-http_secure_link_module
--with-http_slice_module --with-http_ssl_module --with-http_stub_status_module
--with-http_sub_module --with-http_v2_module --with-http_xslt_module=dynamic
--with-ipv6 --with-mail --with-mail_ssl_module --with-md5-asm --with-pcre-jit
--with-sha1-asm --with-stream --with-stream_ssl_module --with-threads
--with-stream --with-stream_ssl_preread_module
```
* etcd version, if have (cmd: run `curl
http://127.0.0.1:9090/v1/server_info` to get the info from server-info API):
* luarocks version, if the issue is about installation (cmd: `luarocks
--version`): v3.4.14
### Minimal test code / Steps to reproduce the issue
I failed to reproduce this problem with a minimal test env, it occurs in
production environment WHEN the concurrent connections are very high - maybe
more than 30000.
the apisix config in our production env(I mocked some id/key):
```shell
curl -X PUT http://localhost:9180/apisix/admin/upstreams/xxx -H 'X-API-KEY:
xxx' --data-raw \
'{
"nodes": [
{
"port": 8080,
"priority": 0,
"weight": 1,
"host": "172.11.160.26"
}
],
"retries": 0,
"scheme": "http",
"pass_host": "pass",
"hash_on": "vars",
"type": "roundrobin"
}'
curl -X PUT http://localhost:9180/apisix/admin/services/xxx -H 'X-API-KEY:
xxx' --data-raw \
'{
"plugins": {
"limit-conn": {
"default_conn_delay": 0.1,
"burst": 0,
"rejected_code": 529,
"conn": 32000,
"key": "server_addr"
},
"limit-req": {
"key": "server_addr",
"rate": 25000,
"burst": 50000,
"rejected_code": 519
},
"prometheus": {}
},
"upstream_id": "xxx",
}'
curl -X PUT http://localhost:9180/apisix/admin/routes/xxx -H 'X-API-KEY:
xxx' --data-raw \
'{
"host": "eks-test",
"uri": "/*",
"upstream_id": "xxx"
}'
```
### What's the actual result? (including assertion message & call stack if
applicable)
apisix retryed some requests.
### What's the expected result?
apisix should not retry any requests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
