sssomeshhh opened a new issue, #13662:
URL: https://github.com/apache/apisix/issues/13662
### Current Behavior
For **native Anthropic (Messages API) upstreams in streaming mode**,
`ai-proxy` reports the normalized token usage `total_tokens` as **output tokens
only** — the prompt/input tokens are silently excluded. The individual
`prompt_tokens` and `completion_tokens` values remain correct; only the
pre-summed `total_tokens` field is wrong.
An Anthropic streaming response splits usage across events: `message_start`
carries `input_tokens` (and an initial `output_tokens`), and `message_delta`
carries the final `output_tokens` but **no** `input_tokens`. The bug is a
two-part interaction:
1. **Tainted per-event total.** In
`apisix/plugins/ai-protocols/anthropic-messages.lua` the `message_delta` branch
builds a per-event total from both fields:
https://github.com/apache/apisix/blob/3.17.0/apisix/plugins/ai-protocols/anthropic-messages.lua#L75-L85
```lua
total_tokens = (data.usage.input_tokens or 0)
+ (data.usage.output_tokens or 0),
```
Because a `message_delta` omits `input_tokens`, this collapses to
**output-only** for that event.
2. **Last-positive-writer-wins overwrite.** `merge_usage` in
`apisix/plugins/ai-providers/base.lua` overwrites every numeric field `> 0`,
and never recomputes `total_tokens` from the merged parts:
https://github.com/apache/apisix/blob/3.17.0/apisix/plugins/ai-providers/base.lua#L71-L92
```lua
for k, v in pairs(parsed.usage) do
if type(v) == "number" and v > 0 then
ctx.ai_token_usage[k] = v
end
end
```
The correct total seeded by `message_start` (input + output) is clobbered
by the output-only total from `message_delta`. `prompt_tokens` survives because
the delta's `input_tokens = 0` fails the `v > 0` guard; `total_tokens` does not
survive because the delta's output-only total is `> 0` and overwrites it.
**Numeric trace** (input = 100, output = 50):
| Event | parsed.usage | ctx.ai_token_usage after merge |
|-------|--------------|--------------------------------|
| `message_start` (`input=100, output=1`) | `{prompt=100, completion=1,
total=101}` | `{prompt=100, completion=1, total=101}` (seed) |
| `message_delta` (`output=50`, no input) | `{prompt=0, completion=50,
total=50}` | `{prompt=100, completion=50, total=50}` |
| `message_stop` | (no usage) | unchanged |
Final `total_tokens = 50`. Correct total = `prompt_tokens +
completion_tokens = 100 + 50 = 150`. The undercount equals the **entire input
token count** on every affected request, so it scales with prompt size (~99%
undercount on large-context / RAG / agentic prompts).
**Concrete downstream impact — `ai-rate-limiting`.** The corrupted
`total_tokens` is consumed on the default configuration. `ai-rate-limiting`'s
`get_token_usage` directly indexes `usage[conf.limit_strategy]`, and the schema
default for `limit_strategy` is `total_tokens`, so the plugin charges the
output-only figure (50) against the token bucket instead of 150. Because the
deficit is the full input token count, a caller can sustain effectively
unbounded input throughput under a `total_tokens` limit — a token quota /
rate-limit bypass on default settings.
**Scope / non-affected paths:**
- Not affected: **non-streaming** Anthropic (usage is summed in one shot),
and **OpenAI-chat / Bedrock-converse / OpenAI-responses** streaming (each emits
a single self-complete usage event with a correct total).
- Not affected: `ai-rate-limiting` with `limit_strategy = prompt_tokens` or
`completion_tokens` (those individual fields are correct), or `limit_strategy =
expression` (reads raw provider fields).
- Not affected: consumers that **sum** the exported nginx vars
`$llm_prompt_tokens + $llm_completion_tokens` (both are correct). Only the
normalized `total_tokens` field is corrupted.
**Relationship to `master`:** the `base.lua` half is already mitigated on
`master` — `merge_usage` now recomputes `total = prompt + completion` and takes
the max (added in PR #13477 / `bf9091acd`):
https://github.com/apache/apisix/blob/a2d34cd62a0c8c8981acfa13f3a6967832569ebd/apisix/plugins/ai-providers/base.lua#L71-L99
That change was **not** shipped in the 3.17.0 release cut, so 3.17.0 is
fully unmitigated. Note the `anthropic-messages.lua` `message_delta`
construction is **unchanged on `master`** (it still omits `input_tokens` from
its per-event total) — it is only masked there by the downstream recompute.
### Expected Behavior
For native Anthropic streaming responses, the normalized `total_tokens`
should equal `prompt_tokens + completion_tokens` (input + output), i.e. `150`
for the trace above — matching non-streaming Anthropic and all other providers.
`ai-rate-limiting` with the default `limit_strategy = total_tokens` should
charge the full input + output token count.
### Error Logs
No error is raised. The output-only total silently overwrites the correct
total in `ctx.ai_token_usage`; there is no log line or warning.
### Steps to Reproduce
1. Run APISIX `3.17.0`.
2. Create a Route with `ai-proxy` pointing at a native **Anthropic Messages
API** upstream, plus `ai-rate-limiting` using its default token strategy
(`limit_strategy = total_tokens`) with a small `limit`, e.g.:
```json
{
"id": "anthropic-messages-stream",
"uri": "/v1/messages",
"methods": ["POST"],
"plugins": {
"ai-proxy": {
"provider": "anthropic",
"auth": { "header": { "x-api-key": "<ANTHROPIC_API_KEY>",
"anthropic-version": "2023-06-01" } },
"options": { "model": "claude-3-5-sonnet-20241022" }
},
"ai-rate-limiting": {
"limit": 10000,
"time_window": 60,
"limit_strategy": "total_tokens"
}
}
}
```
3. Send a **streaming** request (`"stream": true`) with a large prompt
(input ≈ 100 tokens) and a short completion (output ≈ 50 tokens).
4. Observe the token count charged against the `ai-rate-limiting` bucket: it
decrements by the **output tokens only** (~50), not by input + output (~150).
Repeated large-prompt requests are each undercounted by their full input token
count, so far more requests pass the `total_tokens` limit than the real budget
allows.
(Cross-check: the individually exported `$llm_prompt_tokens` and
`$llm_completion_tokens` nginx vars are both correct — isolating the defect to
the summed `total_tokens`.)
### Environment
- APISIX version (run `apisix version`): **3.17.0** (defect introduced with
the `ai-protocols` / `ai-providers` layout in the 3.17.0 line; not present in
3.16.0 or earlier)
- Operating system (run `uname -a`): _<any>_
- OpenResty / Nginx version (run `openresty -V` or `nginx -V`): _<from
`apache/apisix:3.17.0` image>_
- etcd version, if relevant (run `curl
http://127.0.0.1:9090/v1/server_info`): _n/a_
- APISIX Dashboard version, if relevant: _n/a_
- Plugin runner version, for issues related to plugin runners: _n/a_
- LuaRocks version, for installation issues (run `luarocks --version`): _n/a_
### Suggested Fix
Either fix alone stops the bypass; doing both is cleanest:
1. **Backport the `master` recompute** (primary — this is exactly what
`master` already does via PR #13477). After the per-key merge loop in
`merge_usage` (`apisix/plugins/ai-providers/base.lua`), recompute the total
from the accumulated parts rather than carrying a per-event total:
```lua
local computed = (ctx.ai_token_usage.prompt_tokens or 0)
+ (ctx.ai_token_usage.completion_tokens or 0)
if computed > (ctx.ai_token_usage.total_tokens or 0) then
ctx.ai_token_usage.total_tokens = computed
end
```
2. **Stop minting the misleading per-event total at the source.** In
`apisix/plugins/ai-protocols/anthropic-messages.lua`, do not emit a
`total_tokens` from the `message_delta` branch (leave it absent so it can't
pass the `v > 0` guard and overwrite the seeded total); derive the total once
from the merged `prompt_tokens + completion_tokens`. This also removes the
latent/masked defect that still exists on `master`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
