Lewisyixin opened a new issue, #11897:
URL: https://github.com/apache/apisix/issues/11897

   ### Current Behavior
   
   When modify nodes weight, error log will be recorded " all upstream nodes is 
unhealthy"
   
   
   ### Expected Behavior
   
   no error log
   
   ### Error Logs
   
   
![image](https://github.com/user-attachments/assets/2d52c082-098f-4bfe-820c-a7e075001233)
   
   `2025/01/07 18:04:23 [error] 6714#6714: *276732 [lua] balancer.lua:83: 
fetch_health_nodes(): failed to get health check target status, addr: 
10.58.94.168:80, host: nil, err: target not found, client: 127.0.0.1, server: 
_, request: "GET /l HTTP/1.1", host: "l.com"
   2025/01/07 18:04:23 [error] 6714#6714: *276732 [lua] balancer.lua:83: 
fetch_health_nodes(): failed to get health check target status, addr: 
10.58.32.145:80, host: nil, err: target not found, client: 127.0.0.1, server: 
_, request: "GET /l HTTP/1.1", host: "l.com"
   2025/01/07 18:04:23 [warn] 6714#6714: *276732 [lua] balancer.lua:89: 
fetch_health_nodes(): all upstream nodes is unhealthy, use default, client: 
127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com"
   2025/01/07 18:04:24 [error] 6716#6716: *276683 [lua] balancer.lua:83: 
fetch_health_nodes(): failed to get health check target status, addr: 
10.58.32.145:80, host: nil, err: target not found, client: 127.0.0.1, server: 
_, request: "GET /l HTTP/1.1", host: "l.com"
   2025/01/07 18:04:24 [error] 6716#6716: *276683 [lua] balancer.lua:83: 
fetch_health_nodes(): failed to get health check target status, addr: 
10.58.94.168:80, host: nil, err: target not found, client: 127.0.0.1, server: 
_, request: "GET /l HTTP/1.1", host: "l.com"
   2025/01/07 18:04:24 [warn] 6716#6716: *276683 [lua] balancer.lua:89: 
fetch_health_nodes(): all upstream nodes is unhealthy, use default, client: 
127.0.0.1, server: _, request: "GET /l HTTP/1.1", host: "l.com"
   `
   
   ### Steps to Reproduce
   
   1. create a route and upstream with 2 or more nodes. Upstream need turn on 
healthcheck
   
![image](https://github.com/user-attachments/assets/f403c80c-30b2-4f6e-b5b1-2c65799b5d81)
   
![image](https://github.com/user-attachments/assets/fd5c7c5e-85ea-4d4a-a175-740b25898788)
   2. run a wrk command to  make requests to the server continuously.
   
![image](https://github.com/user-attachments/assets/f06004ac-1301-4297-9d08-8cbdeeb681f9)
   3. change node weight  continuously(such as keep executing the following 
command)
   `curl -XPATCH -s -H "x-api-key: $token" 
http://192.168.20.226:9180/apisix/admin/upstreams/547982406942985081 -d 
'{"nodes":{"10.58.32.145:80": 20}}'`
   4. and we will get error log
   
![image](https://github.com/user-attachments/assets/dcacdf86-d73f-4755-860a-84ef10f44e78)
   5.In extreme cases, client will get a 500 internal error(but this cannot be 
reproduced stably)
   
![image](https://github.com/user-attachments/assets/d1de3bb1-a257-4160-83a5-9d6794047ec7)
   
   
   ### Environment
   
   - APISIX version (run `apisix version`): 3.5
   - Operating system (run `uname -a`): 3.10.0-514.26.2.el7.x86_64
   - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): 
openresty/1.21.4.2
   - etcd version, if relevant (run `curl 
http://127.0.0.1:9090/v1/server_info`): 3.5
   - APISIX Dashboard version, if relevant:
   - Plugin runner version, for issues related to plugin runners:
   - LuaRocks version, for installation issues (run `luarocks --version`): 2.3.0
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to