ryan4yin opened a new issue, #7919: URL: https://github.com/apache/apisix/issues/7919
### Current Behavior APISIX's P99 latency is too high.  ### Expected Behavior APISIX's P99 latency should not be so high. ### Error Logs the cpu flame graph I captured using openresty-xray shows the prometheus consumed too much times in the request lifetime:  ### Steps to Reproduce first, create an upstream using the following config: ```json { "id": "xxx-extractor", "desc": "xxx-extractor", "scheme": "http", "type":"roundrobin", "nodes": [ { "host": "172.22.33.44", "port": 8106, "weight": 0, "priority": 0}, { "host": "172.22.33.45", "port": 8106, "weight": 0, "priority": 0}, { "host": "172.22.33.46", "port": 8106, "weight": 0, "priority": 0}, { "host": "172.22.33.47", "port": 8106, "weight": 0, "priority": 0}, { "host": "172.22.33.48", "port": 8106, "weight": 0, "priority": 0} ], "retries": 0, "timeout": { "connect":15, "send":15, "read":15 }, "checks": { "active": { "timeout": 3, "http_path": "/", "host": "xxx.xxx", "healthy": { "interval": 3, "successes": 3 }, "unhealthy": { "interval": 3, "http_failures": 3 }, "req_headers": ["User-Agent: curl/7.29.0"] }, "passive": { "healthy": { "http_statuses": [200, 201], "successes": 3 }, "unhealthy": { "http_statuses": [500, 502, 503, 504], "http_failures": 3, "tcp_failures": 3 } } } } ``` then create a route with the following config: ```yaml { "id": "xxx-ingress", "uri": "/*", "hosts": [ "xxx.xxx" ], "methods": [ "PUT", "GET", "POST", "HEAD" ], "plugins": { "prometheus": {}, "proxy-mirror": { "host": "http://my-app.test.com"; } }, "upstream_id": "xxx-extractor" } ``` and then given about 30 QPS to the APISIX instance, high P99 latency problem will occurs. ### Environment running in Kubernetes, using helm chart `apisix/apisix` with version `0.11.0`. - APISIX version (run `apisix version`): `docker.io/apache/apisix:2.15.0-alpine` - Operating system (run `uname -a`): `Linux apisix-edge-765f88c49f-8tslz 5.4.190-107.353.amzn2.x86_64 #1 SMP Wed Apr 27 21:16:35 UTC 2022 x86_64 Linux` - Host OS Version: `Linux ip-xx-xx-xx-xx.ec2.internal 5.4.204-113.362.amzn2.x86_64 #1 SMP Wed Jul 13 21:34:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux` - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): `docker.io/bitnami/etcd:3.5.4-debian-11-r22` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
