kayx23 commented on code in PR #14097:
URL: https://github.com/apache/apisix/pull/14097#discussion_r4451770888
##########
apisix/plugins/ai-rate-limiting.lua:
##########
@@ -77,9 +77,12 @@ local schema = {
minLength = 1,
description = "Lua arithmetic expression for dynamic token cost
calculation. "
.. "Variables are injected from the LLM API raw usage response
fields. "
- .. "Missing variables default to 0. "
+ .. "Missing variables default to 0, so use the usage field
names "
+ .. "of the API format that the route's clients use. "
Review Comment:
Here `llm_raw_usage` is not always in the client API format. In the
non-streaming path, `base.lua` converts the response and then calls
`client_proto.extract_usage`; in `parse_streaming_response`, it calls
`target_proto.parse_sse_event` and `merge_usage(ctx, parsed)` while the
converter only emits downstream chunks. For an Anthropic client converted to an
OpenAI Chat upstream, non-streaming expressions therefore see Anthropic fields,
while streaming expressions see `prompt_tokens`/`completion_tokens`. The same
`cost_expr` can count one mode and silently count zero in the other. Please
either make `llm_raw_usage` consistently use the client-facing format before
documenting this rule, or qualify the schema/docs to cover the streaming
conversion behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
