shihaiyang-world opened a new issue, #9313: URL: https://github.com/apache/apisix/issues/9313
### Description I have a high-concurrency requirement that requires apisix to support 200,000+ concurrent requests. However, in reality, when the upstream of apisix is set to "discovery_type: kubernetes", apisix becomes a performance bottleneck. On the other hand, when the upstream of apisix is set to "nodes", the performance meets expectations. Here are the steps of the experiment: - Step 1: Prepare a target service - an nginx service, use wrk to perform load testing, and verify that it can support `28W QPS `; this serves as the **baseline**. **( wrk -> nginx[15Pod] )** ✅ - Step 2: Perform load testing on 1 apisix Pod with `8C 8GB`. The result is `8W QPS` , and the CPU usage has already reached 8 cores. At this point, the bottleneck is on the single node of apisix. **( wrk -> apisix[1 Pod] -> nginx[15Pod] )** ✅ - Step 3: Scale apisix to 4 pods and perform load testing. It is expected to reach `28W QPS` , but it did not meet expectations. The test result only reached `170,000 QPS`, and CPU usage did not reach its peak. **(wrk -> apisix[4Pod][apisix discovery] -> nginx[15 Pod])** ❌ - Step 4: Modify the upstream of apisix to "**service IP**", that is, use Kubernetes service discovery. It is expected to reach 28W concurrent requests, and the result meets expectations with a test result of `28W QPS`. **(wrk -> apisix[4Pod][kubernetes discovery] -> nginx[15Pod])** ✅ The screenshot below shows the test result:  **`Why did the performance not meet expectations when using "discovery_type: kubernetes" in the upstream?`** Here's your configuration file for steps 1-3: ```yaml uri: /test-nginx host: nginx-benchmark.com upstream: timeout: connect: 50 send: 50 read: 50 type: roundrobin scheme: http discovery_type: kubernetes pass_host: pass name: nginx service_name: infrafe/devops-test-nginx:http keepalive_pool: idle_timeout: 60 requests: 100000 size: 320 status: 1 ``` And here's your configuration file for step 4: ```yaml uri: /test-nginx host: nginx-benchmark.com upstream: nodes: - host: 10.31.241.215 port: 80 weight: 1 timeout: connect: 6 send: 6 read: 6 type: roundrobin scheme: http pass_host: pass keepalive_pool: idle_timeout: 60 requests: 100000 size: 320 status: 1 ``` ### Environment - APISIX version (run `apisix version`): 2.15.0 - Operating system (run `uname -a`): Linux apisix-rta-68c56cbd58-4dbxt 5.4.119-19-0009.11 #1 SMP Wed Oct 5 18:41:07 CST 2022 x86_64 Linux - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): openresty/1.21.4.1 - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: 2.13 - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
