jackiechan1981 opened a new issue, #13474:
URL: https://github.com/apache/apisix/issues/13474
### Description
I am deployed Apache APISIX on my Ubuntu system and using the ai-proxy
plugin as an LLM router. During operation, I encountered an intermittent
problem: "openai-base.lua:366: failed to connect to LLM server: timeout"
appears periodically, but then recovers after some time. I configured various
configurations from different LLM manufacturers in the routing list, including
Doubao, Qwen, and DeepSeek. This problem occurred for each manufacturer's
model. However, directly using the official base URL provided by the LLM
vendors works without any problems.
After extensive analysis, I haven't been able to identify the root cause.
Any help from the community would be greatly appreciated!
The partial log when requesting the LLM is as follows：
`2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua]
trusted-addresses.lua:46: is_trusted(): trusted_addresses_matcher is not
initialized, client: 10.32.10.24, server: _, request: "POST
/v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxxx.com:9080",
request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua]
radixtree_host_uri.lua:161: match(): route match mode: radixtree_host_uri,
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] init.lua:749:
http_access_phase(): matched route:
{"clean_handlers":{},"value":{"vars":[["post_arg.model","==","deepseek-v4-flash"]],"priority":0,"create_time":1779525995,"update_time":1780539727,"id":"ai-proxy-deepseek","methods":["POST"],"plugins":{"ai-proxy":{"keepalive":true,"provider":"deepseek","_meta":{},"logging":{"summaries":true,"payloads":true},"auth":{"header":{"Authorization":"Bearer
sk-716997d50f9546eab842924e7e4653da"}},"keepalive_timeout":60000,"keepalive_pool":30,"options":{"model":"deepseek-v4-flash"},"timeout":30000,"ssl_verify":false},"kafka-logger":{"kafka_topic":"ai-proxy-llm","inactive_timeout":5,"meta_format":"default","producer_time_linger":1,"producer_type":"async","name":"kafka
logger","required_acks":1,"cluster_name":1,"producer_batch_num":200,"max_req_body_bytes":524288,"producer_max_buffering":50000,"batch_max_size":1000,"brokers":[{"host":"10.32.10.24","port":9094}],"buffer_duration":60,"_meta":{
},"max_resp_body_bytes":524288,"max_retry_count":0,"timeout":3,"meta_refresh_interval":30,"producer_batch_size":1048576,"include_resp_body":false,"include_req_body":false,"retry_delay":1},"key-auth":{"hide_credentials":false,"header":"Authorization","realm":"key","query":"apikey","_meta":{}}},"name":"deepseek-v4-flash","uri":"/v1/*","status":1},"createdIndex":28,"modifiedIndex":315,"has_domain":false,"orig_modifiedIndex":315,"key":"/apisix/routes/ai-proxy-deepseek"},
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] key-auth.lua:120:
phase_func(): hit key-auth rewrite, client: 10.32.10.24, server: _, request:
"POST /v1/chat/completions HTTP/1.1", host: "ai-gateway-test.xxxx.com:9080",
request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] plugin.lua:767:
merge_consumer_route(): route conf:
{"clean_handlers":{},"value":{"vars":[["post_arg.model","==","deepseek-v4-flash"]],"priority":0,"create_time":1779525995,"update_time":1780539727,"id":"ai-proxy-deepseek","status":1,"plugins":{"ai-proxy":{"auth":{"header":{"Authorization":"Bearer
sk-716997d50f9546eab842924e7e4653da"}},"keepalive":true,"_meta":{},"logging":{"summaries":true,"payloads":true},"timeout":30000,"keepalive_timeout":60000,"keepalive_pool":30,"options":{"model":"deepseek-v4-flash"},"provider":"deepseek","ssl_verify":false},"kafka-logger":{"kafka_topic":"ai-proxy-llm","inactive_timeout":5,"meta_format":"default","_meta":{},"name":"kafka
logger","producer_max_buffering":50000,"required_acks":1,"cluster_name":1,"producer_batch_num":200,"max_req_body_bytes":524288,"retry_delay":1,"batch_max_size":1000,"brokers":[{"port":9094,"host":"10.32.10.24"}],"buffer_duration":60,"producer_type":"async","max_resp_bod
y_bytes":524288,"producer_batch_size":1048576,"timeout":3,"meta_refresh_interval":30,"max_retry_count":0,"include_resp_body":false,"include_req_body":false,"producer_time_linger":1},"key-auth":{"hide_credentials":false,"header":"Authorization","realm":"key","_meta":{},"query":"apikey"}},"name":"deepseek-v4-flash","uri":"/v1/*","methods":["POST"]},"has_domain":false,"modifiedIndex":315,"orig_modifiedIndex":315,"key":"/apisix/routes/ai-proxy-deepseek","createdIndex":28},
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] plugin.lua:768:
merge_consumer_route(): consumer group conf: null, client: 10.32.10.24, server:
_, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] init.lua:825:
http_access_phase(): find consumer model-test, config changed: true, client:
10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] openai-base.lua:285:
request extra_opts to LLM server:
{"conf":{},"auth":{"header":{"Authorization":"Bearer
sk-xxxxx"}},"model_options":{"model":"deepseek-v4-flash"}}, client:
10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] openai-base.lua:362:
sending request to LLM server:
{"body":{"model":"deepseek-v4-flash","max_tokens":16,"messages":[{"role":"user","content":"Reply
with exactly one word:
OK"}]},"query":{},"path":"/chat/completions","method":"POST","host":"api.deepseek.com","scheme":"https","headers":{"X-Consumer-Username":"model-test","Content-Type":"application/json","x-user-id":"13662621035","x-forwarded-proto":"http","content-type":"application/json","x-forwarded-host":"ai-gateway-test.xxxx.com","x-forwarded-port":"9080","connection":"keep-alive","user-agent":"Java/17.0.17","accept":"text/html,
image/gif, image/jpeg, */*; q=0.2","authorization":"Bearer
sk-xxxxx"},"port":443,"ssl_server_name":"api.deepseek.com","ssl_verify":false},
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] client.lua:123:
dns_parse(): dns resolve api.deepseek.com, result:
{"ttl":304,"address":"192.12.94.30","name":"api.deepseek.com","class":1,"section":3,"type":1},
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] resolver.lua:84:
parse_domain(): parse addr:
{"ttl":304,"name":"api.deepseek.com","section":3,"address":"192.12.94.30","class":1,"type":1},
client: 10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1",
host: "ai-gateway-test.xxxx.com:9080", request_id:
"206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] resolver.lua:85:
parse_domain(): resolver: ["10.32.40.1","10.32.40.2"], client: 10.32.10.24,
server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] resolver.lua:86:
parse_domain(): host: api.deepseek.com, client: 10.32.10.24, server: _,
request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:45:56 [info] 24472#24472: *1280418 [lua] resolver.lua:88:
parse_domain(): dns resolver domain: api.deepseek.com to 192.12.94.30, client:
10.32.10.24, server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:46:02 [info] 24475#24475: *1280546 [lua] v3.lua:189:
server_version(): v3 request uri: /version, timeout: 30, context: ngx.timer
2026/06/04 13:46:02 [info] 24475#24475: *1280546 [lua] v3.lua:76:
choose_endpoint(): choose endpoint: http://10.32.10.24:2379, context: ngx.timer
2026/06/04 13:46:02 [info] 24475#24475: *1280546 [lua] v3.lua:189:
_request_uri(): v3 request uri: /kv/range, timeout: 30, context: ngx.timer
2026/06/04 13:46:02 [info] 24475#24475: *1280546 [lua] v3.lua:76:
choose_endpoint(): choose endpoint: http://10.32.10.24:2379, context: ngx.timer
2026/06/04 13:46:17 [info] 24475#24475: *1280879 [lua] v3.lua:189:
server_version(): v3 request uri: /version, timeout: 30, context: ngx.timer
2026/06/04 13:46:17 [info] 24475#24475: *1280879 [lua] v3.lua:76:
choose_endpoint(): choose endpoint: http://10.32.10.24:2379, context: ngx.timer
2026/06/04 13:46:17 [info] 24475#24475: *1280879 [lua] v3.lua:189:
_request_uri(): v3 request uri: /kv/range, timeout: 30, context: ngx.timer
2026/06/04 13:46:17 [info] 24475#24475: *1280879 [lua] v3.lua:76:
choose_endpoint(): choose endpoint: http://10.32.10.24:2379, context: ngx.timer
2026/06/04 13:46:26 [error] 24472#24472: *1280418 [lua] openai-base.lua:366:
failed to connect to LLM server: timeout, client: 10.32.10.24, server: _,
request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:46:26 [warn] 24472#24472: *1280418 [lua] plugin.lua:1224:
common_phase(): ai-proxy exits with http status code 504, client: 10.32.10.24,
server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"
2026/06/04 13:46:26 [info] 24472#24472: *1280418 [lua] log-util.lua:297:
get_log_entry(): metadata: null while logging request, client: 10.32.10.24,
server: _, request: "POST /v1/chat/completions HTTP/1.1", host:
"ai-gateway-test.xxxx.com:9080", request_id: "206f8f3b0ec5cd8ed43fd4dfff7fb2ea"`
### Environment
- APISIX version (run `apisix version`): 3.16.0
- Operating system (run `uname -a`): Ubuntu 24.04.3
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`):
openresty/1.27.1.2
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): etcd Version: 3.5.4
- APISIX Dashboard version, if relevant:
- Plugin runner version, for issues related to plugin runners:
- LuaRocks version, for installation issues (run `luarocks --version`):
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
