shreemaan-abhishek opened a new pull request, #13778:
URL: https://github.com/apache/apisix/pull/13778
### Description
`ai-proxy`, `ai-proxy-multi` and `ai-request-rewrite` all send their outbound
LLM requests through `apisix/plugins/ai-transport/http.lua`. This makes that
transport prefer `ngx_http_ffi_client`, an HTTP client implemented as an
nginx C
module, and keeps `lua-resty-http` as the fallback.
`ngx_http_ffi_client` exposes the same object API as `lua-resty-http`
(`new`, `set_timeout`, `connect`, `request`, `res.body_reader`,
`res:read_body`, `set_keepalive`, `close`) and does the HTTP/1.1 framing in
C,
which costs roughly a third of the outbound CPU time on this path.
The module only exists when the runtime was built with it, so the transport
resolves the client once per worker and falls back when it is absent. **On a
runtime without the module nothing changes**: the same `lua-resty-http` code
runs as before.
`plugin_attr.ai-proxy.http_client` pins the choice for operators who want it
fixed either way:
```yaml
plugin_attr:
ai-proxy:
http_client: auto # auto (default), ffi, or lua-resty-http
```
One behaviour change applies to both clients: `Connection` and
`Transfer-Encoding` are now dropped from the headers forwarded to the LLM
upstream, alongside `Host`, `Content-Length` and `Accept-Encoding`. Both are
hop-by-hop headers describing the downstream connection. The client frames
the
request body with `Content-Length` itself, so forwarding a client's
`Transfer-Encoding: chunked` put two conflicting framing headers on the wire,
and forwarding `Connection: close` desynchronised the upstream connection and
its keepalive pool.
#### Which issue(s) this PR fixes:
N/A
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not,
please discuss on the [APISIX mailing
list](https://github.com/apache/apisix/tree/master#community) first)
### Test plan
`t/plugin/ai-transport-http.t` gains four cases covering client selection:
the
FFI client is used when present, the transport falls back for the rest of the
worker's life when the module's Lua half loads but the C module is not built
in,
`plugin_attr` pins `lua-resty-http` over an available FFI client, and
hop-by-hop
headers are dropped from the forwarded set.
Beyond the plugin tests, the client itself was exercised against a real
build:
an OpenResty carrying the module, driven through the exact call sequence this
transport uses, covering a buffered request, an SSE stream read through
`body_reader`, `set_keepalive`, `close`, and the error strings `handle_error`
maps to 502/504.
`t/plugin/ai-proxy*.t` and `t/plugin/ai-transport-*.t` were run before and
after
the change with identical results.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
