xuande commented on issue #12710:
URL: https://github.com/apache/apisix/issues/12710#issuecomment-3461387586
It works now. With the prometheus plugin added to this route, `llm_*_tokens`
metrics are being collected normally.
Route configuration below:
```
{
"list": [
{
"key": "/apisix/routes/00000000000000000045",
"modifiedIndex": 433,
"value": {
"uri": "/anythings",
"plugins": {
"ai-proxy": {
"options": {
"model": "DeepSeek-R1-Distill-Qwen-7B"
},
"auth": {
"header": {
"Authorization": "Bearer token-123"
}
},
"logging": {
"summaries": true
},
"provider": "openai-compatible",
"override": {
"endpoint":
"http://127.0.0.1:5111/v1/chat/completions";
}
},
"file-logger": {
"path": "/usr/local/apisix/logs/api-access.log",
"max_size": 100,
"format": {
"client_ip": "$remote_addr",
"route_id": "$route_id",
"host": "$host",
"method": "$request_method",
"uri": "$uri",
"request_time": "$request_time",
"status": "$status"
},
"max_age": 7,
"compress": true,
"max_backups": 3
},
"key-auth": {},
"prometheus": {}
},
"methods": [
"POST"
],
"create_time": 1761290259,
"update_time": 1761656372,
"id": "00000000000000000045"
},
"createdIndex": 46
}
],
"total": 1
}
```
After the modification, the `llm_*_tokens` information can now be collected
normally.
```
...
apisix_http_latency_sum{type="request",route="00000000000000000045",service="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_chat",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
1338.9999866486
apisix_http_latency_sum{type="request",route="00000000000000000045",service="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_stream",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
3951.9999027252
# HELP apisix_http_requests_total The total number of client requests since
APISIX started
# TYPE apisix_http_requests_total gauge
apisix_http_requests_total 207
# HELP apisix_http_status HTTP status codes per service in APISIX
# TYPE apisix_http_status counter
apisix_http_status{code="200",route="00000000000000000045",matched_uri="/anythings",matched_host="",service="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_chat",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
1
apisix_http_status{code="200",route="00000000000000000045",matched_uri="/anythings",matched_host="",service="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_stream",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
3
# HELP apisix_llm_active_connections Number of active connections to LLM
service
# TYPE apisix_llm_active_connections gauge
apisix_llm_active_connections{route="",route_id="00000000000000000045",matched_uri="/anythings",matched_host="",service="",service_id="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_chat",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
0
apisix_llm_active_connections{route="",route_id="00000000000000000045",matched_uri="/anythings",matched_host="",service="",service_id="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_stream",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
0
# HELP apisix_llm_completion_tokens LLM service consumed completion tokens
# TYPE apisix_llm_completion_tokens counter
apisix_llm_completion_tokens{route_id="00000000000000000045",service_id="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_chat",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
100
apisix_llm_completion_tokens{route_id="00000000000000000045",service_id="",consumer="test",node="ai-proxy-openai-compatible",request_type="ai_stream",request_llm_model="",llm_model="DeepSeek-R1-Distill-Qwen-7B"}
300
....
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
