This is an automated email from the ASF dual-hosted git repository.
hsluoyz pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/casbin-gateway.git
The following commit(s) were added to refs/heads/master by this push:
new 62ce86c fix(routing): keep conversation stickiness from stepping over
a rule
62ce86c is described below
commit 62ce86c9c0204a304b1050eb338906fa5c7292a6
Author: Yang Luo <[email protected]>
AuthorDate: Sat Oct 31 18:47:35 2026 +0800
fix(routing): keep conversation stickiness from stepping over a rule
---
README.md | 2 +-
README_zh.md | 2 +-
object/provider_pool.go | 47 +++++++++++++++++++++++++++++++++++++----------
3 files changed, 39 insertions(+), 12 deletions(-)
diff --git a/README.md b/README.md
index d6df754..32fba54 100644
--- a/README.md
+++ b/README.md
@@ -319,7 +319,7 @@ Bind one of them to an agent and pick the others as its
fallbacks, and the agent
| Rotate | the next provider, for each new conversation |
| Most left first | the plan with the most left of its tightest window; API
keys come after every plan |
-Whatever the strategy, a conversation stays on the provider that answered it
until it has been idle for 15 minutes, so the prompt cache it built there is
spent rather than paid for again elsewhere. Gateway knows a conversation by the
session id the agent sends, which Claude Code and Codex do, and otherwise by
how it began, which every later turn repeats. A plan that has run out of a
window waits behind the others until that window resets, rather than for the
few minutes a cooldown lasts, [...]
+Whatever the strategy, a conversation stays on the provider that answered it
until it has been idle for 15 minutes, so the prompt cache it built there is
spent rather than paid for again elsewhere. It chooses only among the providers
the request's own routing offers, so a rule's first step is never stepped over
by a conversation that was answered somewhere else before. Gateway knows a
conversation by the session id the agent sends, which Claude Code and Codex do,
and otherwise by how it [...]
**What each plan has left** is on its card on the Providers page: the 5-hour
and the weekly window of a Claude or ChatGPT plan, the premium requests of a
Copilot one, each with when it resets. Gateway reads it with the sign-in
itself, and keeps it current from the headers of every answer it relays. When
most of a window is about to be lost to its reset - half of it or more left,
with an hour to go, or a day for the weekly one - the card says so.
diff --git a/README_zh.md b/README_zh.md
index 67d3b3d..43731f6 100644
--- a/README_zh.md
+++ b/README_zh.md
@@ -316,7 +316,7 @@ Key 没余额了、中转站挂了、厂商在会话中途开始限流 —— Ga
| 轮流 | 下一个 Provider，每个新对话轮一次 |
| 先用剩余多的 | 最紧的那个窗口剩得最多的订阅；API Key 排在所有订阅之后 |
-不管哪种策略，一个对话都会留在回答它的 Provider 上，直到空闲 15
分钟，这样在那里建起来的提示词缓存能接着用，而不是换一家重新付钱。Gateway 按 Agent 发来的会话 ID 认出一个对话（Claude Code 和
Codex
都会发），没有的话就看它是怎么开头的，后面每一轮都会重复开头。某个窗口额度用完的订阅会排到最后，直到这个窗口重置，而不是只冷却几分钟；前面说的切换规则对每个请求照样生效。
+不管哪种策略，一个对话都会留在回答它的 Provider 上，直到空闲 15
分钟，这样在那里建起来的提示词缓存能接着用，而不是换一家重新付钱。但它只在这次请求的路由给出的 Provider
里挑，所以一个规则的第一步不会被之前答过别处的对话跨过去。Gateway 按 Agent 发来的会话 ID 认出一个对话（Claude Code 和
Codex
都会发），没有的话就看它是怎么开头的，后面每一轮都会重复开头。某个窗口额度用完的订阅会排到最后，直到这个窗口重置，而不是只冷却几分钟；前面说的切换规则对每个请求照样生效。
**每个套餐还剩多少**，写在 Providers 页面上它的卡片里：Claude 和 ChatGPT 套餐的 5 小时窗口和每周窗口，Copilot
套餐的高级请求，以及各自什么时候重置。Gateway 用这个登录本身去查，转发的每次回答的响应头也会更新它。一个窗口的大部分额度快要因为重置作废时 ——
还剩一半以上，离重置不到一小时（每周窗口是一天）—— 卡片上会提示。
diff --git a/object/provider_pool.go b/object/provider_pool.go
index 37168f6..4dab33a 100644
--- a/object/provider_pool.go
+++ b/object/provider_pool.go
@@ -224,8 +224,12 @@ func firstNonSystem(messages []json.RawMessage) []byte {
}
// PreferSticky moves the provider that answered this conversation last to the
-// front, so its prompt cache is spent rather than paid for again elsewhere. A
-// provider that is cooling down or out of its plan is left where it is.
+// front of the stretch of attempts it belongs to, so its prompt cache is spent
+// rather than paid for again elsewhere. A stretch is the attempts one routing
+// rule put there, or the ruleless ones below them: a rule is the operator
+// saying where a model goes, and a conversation that has been answered by some
+// other provider before must not step over it. A provider that is cooling down
+// or out of its plan is left where it is.
func PreferSticky(key string, attempts []RouteAttempt) []RouteAttempt {
if key == "" || len(attempts) < 2 {
return attempts
@@ -241,16 +245,39 @@ func PreferSticky(key string, attempts []RouteAttempt)
[]RouteAttempt {
return attempts
}
- front := []RouteAttempt{}
- rest := []RouteAttempt{}
- for _, attempt := range attempts {
- if attempt.providerId() == entry.provider {
- front = append(front, attempt)
- } else {
- rest = append(rest, attempt)
+ ordered := make([]RouteAttempt, 0, len(attempts))
+ for start := 0; start < len(attempts); {
+ end := start + 1
+ for end < len(attempts) && attempts[end].Route ==
attempts[start].Route {
+ end++
+ }
+ ordered = appendSticky(ordered, attempts[start:end],
entry.provider)
+ start = end
+ }
+ return ordered
+}
+
+// appendSticky adds one stretch with the remembered provider moved to its
+// front, and the rest in the order it was given.
+func appendSticky(dst []RouteAttempt, stretch []RouteAttempt, providerId
string) []RouteAttempt {
+ promoted := -1
+ for i, attempt := range stretch {
+ if attempt.providerId() == providerId {
+ promoted = i
+ break
+ }
+ }
+ if promoted < 0 {
+ return append(dst, stretch...)
+ }
+
+ dst = append(dst, stretch[promoted])
+ for i, attempt := range stretch {
+ if i != promoted {
+ dst = append(dst, attempt)
}
}
- return append(front, rest...)
+ return dst
}
// RememberSticky records which provider answered a conversation.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
