Arjen10 opened a new issue, #14199: URL: https://github.com/apache/apisix/issues/14199
### Current Behavior When i am reading the [Prometheus docs](https://apisix.apache.org/zh/docs/apisix/plugins/prometheus/#apisix_llm_active_connections-%E7%9A%84%E6%A0%87%E7%AD%BE), I noticed something strange in the description of `apisix_llm_active_connections`. it says retries can make the value higher than it should be. but `apisix_llm_active_connections` is a gauge, not a counter, so I think this is a bug. ### Expected Behavior Since it’s a gauge, the value should be decreased promptly when a connection is no longer active, even if it fails. It should reflect what “active” means. Otherwise, it can be confusing. ### Error Logs ``` # HELP apisix_llm_active_connections Number of active connections to LLM service # TYPE apisix_llm_active_connections gauge apisix_llm_active_connections{route="",route_id="debug",matched_uri="/debug",matched_host="",service="",service_id="",consumer="",node="debug-a",request_type="ai_chat",request_llm_model="gpt-4",llm_model="gpt-4"} 5 apisix_llm_active_connections{route="",route_id="debug",matched_uri="/debug",matched_host="",service="",service_id="",consumer="",node="debug-b",request_type="ai_chat",request_llm_model="gpt-4",llm_model="gpt-4"} 0 ``` This "debug-a" keep going up and will never stop. it's bad ### Steps to Reproduce 1、create mock route ``` curl --location --request PUT 'http://172.18.0.21:9180/apisix/admin/routes/123-mock' \ --header 'Content-Type: application/json' \ --header 'X-API-KEY: ••••••' \ --data '{ "uri": "/123-mock", "plugins": { "fault-injection": { "abort": { "http_status": 500, "body": "injected upstream failure" } } } }' ``` 2、enable ai-proxy-multi and prometheus ``` curl --location --request PUT 'http://172.18.0.21:9180/apisix/admin/routes/debug' \ --header 'Content-Type: application/json' \ --header 'X-API-KEY: ••••••' \ --data '{ "uri": "/debug", "plugins": { "prometheus": {}, "ai-proxy-multi": { "instances": [ { "name": "debug-a", "provider": "openai", "weight": 1, "priority": 10, "auth": { "header": { "Authorization": "Bearer test-token" } }, "override": { "endpoint": "http://172.18.0.21:9080/123-mock"; } }, { "name": "debug-b", "provider": "openai", "weight": 1, "priority": 0, "auth": { "header": { "Authorization": "Bearer test-token" } }, "override": { "endpoint": "http://172.18.0.21:9080/123-mock"; } } ], "fallback_strategy": [ "http_5xx" ] } } }' ``` 3、 send req ``` curl --location 'http://172.18.0.21:9080/debug' \ --header 'Content-Type: application/json' \ --data '{ "model": "gpt-4", "messages": [ { "role": "user", "content": "hello" } ] }' ``` ### Environment - APISIX version (run `apisix version`): 3.19.0 - Operating system (run `uname -a`): - OpenResty / Nginx version (run `openresty -V` or `nginx -V`): - etcd version, if relevant (run `curl http://127.0.0.1:9090/v1/server_info`): - APISIX Dashboard version, if relevant: - Plugin runner version, for issues related to plugin runners: - LuaRocks version, for installation issues (run `luarocks --version`): -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
