AlinsRan commented on code in PR #13676:
URL: https://github.com/apache/apisix/pull/13676#discussion_r3576301474
##########
apisix/plugins/ai-proxy/schema.lua:
##########
@@ -303,7 +360,35 @@ _M.ai_proxy_multi_schema = {
properties = {
algorithm = {
type = "string",
- enum = { "chash", "roundrobin" },
+ enum = { "chash", "roundrobin", "semantic" },
+ description = "Load-balancing algorithm. Note: 'semantic' "
+ .. "picks an instance by prompt similarity and does
not "
+ .. "participate in health checks or fallback_strategy
/ "
+ .. "retry — an upstream failure on the chosen instance
is "
+ .. "returned to the client. It only falls back (to the
"
+ .. "catchall, else the first instance) when no
instance "
+ .. "clears its threshold or embedding fails.",
+ },
+ threshold = {
+ type = "number",
+ description = "Global minimum cosine similarity for the "
+ .. "semantic algorithm. When no instance clears its "
+ .. "threshold the request falls back to the catchall "
+ .. "instance, or the first instance if none is set.",
+ },
+ aggregation = {
+ type = "string",
+ enum = { "avg", "max", "min" },
Review Comment:
You are right, and the numbers back it up. `examples` are alternative
phrasings of one intent, so the question is "does **any** of them match" — that
is `max`. I ran the aggregation on a concrete case:
| | A: one dead-on example + two broader ones (1.0 / 0.5 / 0.4) | B:
uniformly mediocre (0.7 / 0.7) | picked |
|---|---|---|---|
| `avg` | 0.633 | 0.700 | **B** — despite A having a perfect match |
| `max` | 1.000 | 0.700 | **A** |
| `min` | 0.400 | 0.700 | **B** |
And your "adding a third example breaks the route" point reproduces exactly:
`avg` drops 0.750 → 0.633 when a broader example is appended, so a route that
cleared a 0.7 threshold stops clearing it — for `max` nothing changes.
`min` is worse still: it demands that *every* phrasing match, which is
nonsense for alternatives.
So there is no use case I can defend for either, and `avg`-as-default meant
the out-of-the-box behaviour was the diluted one. Dropped the knob entirely in
9c58718 — no enum, no default: an instance is scored by its best-matching
example. `semantic.aggregate(scores, method)` is now just
`semantic.max(scores)`.
##########
apisix/plugins/ai-providers/azure-openai.lua:
##########
@@ -32,6 +32,7 @@ return require("apisix.plugins.ai-providers.base").new(
path = "/completions",
rewrite_request_body = rewrite_chat_request_body,
},
+ ["openai-embeddings"] = { path = "/embeddings" },
Review Comment:
Confirmed — this was broken, thanks for catching it. `build_request`
resolves `parsed_url.path or opts.target_path`, and an Azure endpoint always
carries a path, so the capability `/embeddings` was dead code. Worse, my doc
example stopped at the deployment root, so following it would have POSTed there
and 404'd — and since every routing test used `provider: openai`, nothing
exercised this.
Went with (a), matching the convention the azure chat path and
`ai-cache-semantic` already use (full URL, `api-version` in the query):
- `azure-openai` now declares the capability with **no** default path
(`["openai-embeddings"] = {}`), so a path-less endpoint fails loudly in
`build_request` instead of silently building a wrong URL.
- `check_schema` rejects an azure `embeddings.endpoint` without a deployment
path — otherwise the route would fail open to the catchall on every request,
i.e. silently never work.
- Docs (en + zh) and the schema description now show the full URL:
`https://{resource}.openai.azure.com/openai/deployments/{deployment}/embeddings?api-version=2024-02-01`.
- Added an end-to-end test that drives a real azure-shaped URL and asserts
the request lands on the deployment path *with* `api-version`, plus a negative
schema test.
I did not hard-require the literal `/embeddings` suffix, only a non-root
path — happy to tighten that if you would rather have it enforced.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
