membphis commented on PR #14028:
URL: https://github.com/apache/apisix/pull/14028#issuecomment-6160285552
A non-blocking accounting observation for the author to decide whether to
address in this PR or in a follow-up:
The converter now recognizes `usage.cache_creation_input_tokens`,
`prompt_tokens_details.cache_creation_tokens`, and
`prompt_tokens_details.cache_write_tokens`, but the converted streaming path
still collects normalized usage through `openai-chat.parse_sse_event`, which
only reads `prompt_tokens_details.cache_creation_input_tokens`.
For example, an Anthropic client with an OpenAI-compatible upstream can
receive this final streaming usage:
```json
{
"prompt_tokens": 100,
"completion_tokens": 10,
"cache_creation_input_tokens": 30,
"prompt_tokens_details": { "cached_tokens": 20 }
}
```
A focused probe of the reviewed Lua modules shows that the converted client
response reports `input_tokens=50`, `cache_read_input_tokens=20`, and
`cache_creation_input_tokens=30`, while the streaming normalized usage reports
`cache_creation_input_tokens=0`. That zero flows into
`$llm_cache_creation_input_tokens`, the access log, and `llm_summary`. The
non-streaming conversion reports 30 correctly. The same mismatch occurs with
the other two newly supported cache-write field aliases. The corrected prompt
and total token counts on this path remain correct.
Relevant code:
[converter](https://github.com/apache/apisix/blob/9775c8415eeab2193128d633bc763310ddc3c4df/apisix/plugins/ai-protocols/converters/anthropic-messages-to-openai-chat.lua#L1042)
and [streaming usage
extraction](https://github.com/apache/apisix/blob/9775c8415eeab2193128d633bc763310ddc3c4df/apisix/plugins/ai-protocols/openai-chat.lua#L115).
A possible fix is to share the cache-write field extraction rules between
conversion and usage normalization, preserving raw usage, and add streaming E2E
assertions for both the response and access-log cache counts for all three
field shapes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
