janiussyafiq commented on issue #13755:
URL: https://github.com/apache/apisix/issues/13755#issuecomment-5275753971
Hi @RGZingYang, this can be achieved using the `ai-rate-limiting` plugin.
It counts the tokens from the LLM response `usage` field against a quota,
and once the quota is exhausted it rejects follow-up requests at the gateway
without forwarding them to the upstream LLM.
Key the quota per user with the `rules` option (APISIX >= 3.16.0) plus an
authentication plugin so `$consumer_name` resolves:
```json
{
"uri": "/chat",
"plugins": {
"key-auth": {},
"ai-proxy": { ... },
"ai-rate-limiting": {
"rules": [
{ "key": "$consumer_name", "count": 100000, "time_window": 2592000 }
],
"rejected_code": 429,
"rejected_msg": "token quota exhausted"
}
}
}
```
Counters are scoped to the resource the plugin is attached to:
- On a **route**: one counter per user per route (your "user-route pair"
case).
- On a shared **service**: one counter per user across all its routes (your
"per user" case).
Notes:
- It only works together with `ai-proxy` / `ai-proxy-multi`.
- Accounting is post-paid (cost is known only after the response), so the
request that crosses the quota still completes; everything after it is blocked.
- For streaming, make sure the provider returns `usage` in the stream
(OpenAI: `"stream_options": {"include_usage": true}`).
For more detailed reference you can read this documentation:
https://apisix.apache.org/docs/apisix/plugins/ai-rate-limiting/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
