AlinsRan opened a new pull request, #13676: URL: https://github.com/apache/apisix/pull/13676
### Description Add a `semantic` load-balancing algorithm to the `ai-proxy-multi` plugin. Instead of selecting an LLM instance by weight (`roundrobin`) or hash (`chash`), it picks the instance whose configured `examples` are semantically closest to the request prompt — enabling intent-based routing such as sending coding prompts to a strong model and cheap/short prompts to a smaller one, all behind a single route with `"model": "auto"`. How it works: - Each non-`catchall` instance declares `examples` (utterances that describe the prompts it should serve). - On the request path, the plugin embeds the instance examples (once per config version, cached) and the incoming prompt via a configurable OpenAI-compatible `embeddings` endpoint, computes cosine similarity, aggregates each instance's per-example scores (`avg`/`max`/`min`), and routes to the highest-scoring instance that clears its threshold (per-instance `threshold`, else `balancer.threshold`). - **Fail-open by design:** when no instance clears its threshold, or embedding fails for any reason (network error, non-200, malformed vector, dimension mismatch), the request falls back to the `catchall` instance (else the first instance). The routing path never returns 500. - No vector database is required — similarity is computed in-process over a small set of reference vectors. New configuration: `balancer.algorithm=semantic`, `balancer.threshold`, `balancer.aggregation`, `balancer.expose_scores`; per-instance `examples` / `threshold` / `catchall`; and an `embeddings` block (`provider`, `model`, `endpoint`, `auth`, `timeout`, `ssl_verify`). Note: the `semantic` algorithm does not participate in health checks or `fallback_strategy` / retry — an upstream failure on the chosen instance is returned to the client. Upstream-failure failover is planned as a follow-up. #### Which issue(s) this PR fixes: Fixes # ### Checklist - [x] I have explained the need for this PR and the problem it solves - [x] I have explained the changes or the new features added to this PR - [x] I have added tests corresponding to this change - [x] I have updated the documentation to reflect this change - [x] I have verified that this change is backward compatible (existing `roundrobin` / `chash` routes are unaffected; all new fields are optional) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
