GitHub user purushah created a discussion: [Discussion] Quality-triggered
escalation for model routing
Hi all. I'd like to propose letting a router escalate to a stronger candidate
when the cheaper model's **answer** is not good enough, not only when the call
**fails**. This is the "cascade" pattern. I want to agree on the shape here
before opening a PR.
## Today
A `MODEL_ROUTER` (#897) picks one candidate per request with rules, an LLM
judge, or a custom executor. With `fallback(true)` the remaining candidates are
tried in declaration order, but only after the selected model **fails**, after
its retries or immediately for a pre-call failure such as an unresolved
resource. The loop does check a response for truncation, a missing structured
output and the like, but a successful answer is final however poor it is, and
tool rounds stay on the selected model. In short: routing decides before the
call, and nothing checks the quality of a successful answer after it.
## How others do it
| Pattern | Decides | Signal | Examples |
|---|---|---|---|
| Predictive routing | before the call | predicted difficulty or quality |
Bedrock Intelligent Prompt Routing, RouteLLM, OpenRouter Auto, LangChain4j
`RoutingChatModel`, Flink Agents' LLM judge (#897) |
| Cascade | after the call | quality of the actual answer | FrugalGPT and
AutoMix (research); OpenRouter's confidence-threshold guide and its
"Jev-verified cascade", where Jev is a decision model (recipes in application
code); built in: Portkey, where a failed output guardrail becomes a status code
that its fallback config acts on, and NVIDIA NeMo Switchyard, where an LLM
judge rates the weak model's completed turn, the buffered reply is discarded
and the strong model serves the same turn |
Gateways (LiteLLM, OpenRouter, Portkey) declare the fallback list explicitly,
cap it at a few models, and trigger on errors; Portkey's guardrail status codes
are the one quality trigger. Every system above judges a complete answer, none
a streamed one. #1324 proposes a LiteLLM Router integration for routing and
failover among deployments; this proposal is about answer quality across models
and is complementary.
Three findings shaped the design:
- **Two tiers are enough.** Bouchard (2026, v2): a second model adds 2.1 to 8.2
points over the best single model at matched budget, averaged across budgets; a
third, fourth or fifth model gave no budget-averaged gain in 118 of 135
comparisons and at most 0.43 points.
- **The signal must be about the answer, not the question.** Same paper,
simulated scores: a score that also encodes shared question difficulty removes
the gain from a third tier even at the same AUROC, because the next model fails
on the same questions; the two-model gain survives.
- **Confidence is not calibrated out of the box.** Self-scores run high.
Decision models report a probability but are not calibrated everywhere: an
independent evaluation of Jev measured expected calibration error of 0.011 on
one task and 0.108 on another. Thresholds come from error rates on your own
traffic.
## Proposal
An optional **escalation policy** next to `fallback`. Fallback keeps handling
errors. Escalation handles answers that pass but score badly.
```java
ModelRouter.of("small", "big")
.fallback(true) // on failure: next
candidate, as today
.escalation(Escalation.tiers("small", "big") // after the answer:
move up?
.when(QualitySignals.answerConfidence("confidence", 0.7)))
.build();
```
Names are illustrative. This example sets no `strategy(...)`, which is required
today; I propose it becomes optional when tiers are set, so every request
starts on the first tier. A strategy may still pick the starting tier. A low
confidence, or a structural problem, re-sends the original request to `big`,
with `big`'s own prompt and tools; the cheap attempt is discarded. `big` is the
last tier, so its answer is final.
**Signals** are checked after each model round and before any tool call from
that round is dispatched, so a discarded attempt has no side effects. The
structural signal is always on; `when` and `or` add signals to it. Responses
the loop already treats as failures (truncation, content filter, missed output
schema) stay on the fallback path, and failure wins when both apply.
1. **Structural** (always on): empty content with no tool call, a call to a
tool the candidate does not have, arguments that fail the tool's input schema,
the same tool call repeated with identical arguments in consecutive rounds.
These are pure checks on the response: new code in the chat action, no model
call, deterministic on replay.
2. **Answer confidence** (opt-in): the candidate's prompt asks for a numeric
`confidence` field. When the request uses structured output the field is added
to the schema and stripped from the returned content; otherwise it is read from
a trailing JSON block. An ordering signal, not a probability. Exact mechanics
are question 5.
3. **Decision model on the answer** (opt-in): a `DECISION_MODEL` (#1264) gets
the request plus the answer and one predicate question; `probability_true`
below the threshold escalates. A direct call under `durableExecute` with id
`escalate:<router>:<candidate>`, not through the decision action, so ids do not
collide. On tool rounds the route is direct, so the router name comes from the
routing metadata parked for the request. Picking the starting tier with a
decision model is #1325.
4. **Custom**: an executor with the request and response, like
`CustomRoutingExecutor`.
**Tiers** are explicit, cheapest first, at least two, each a candidate;
candidates left out stay fallback-only. Tier count bounds the hops,
`maxEscalations(n)` can cap lower. Two tiers, one hop, is the recommended
default. The last tier always accepts on quality. If it fails after its
retries, I propose fallback skips candidates that already answered this
request, which is a change from today's fixed order; what happens with none
left is question 6. Declaration order in `of(...)` keeps meaning fallback order.
**Tool loop:** I propose that once a request escalates it stays on the stronger
model (question 3). Switching back per round breaks prefix caching. Today tool
rounds run under the plain `chat` id with no fallback; that stays as it is.
**Replay:** each initial attempt already has its own durable id,
`chat:<router>:<candidate>`. Confidence and decision-model verdicts are
recorded beside it; structural checks are recomputed.
**Observability and cost:** a `ModelRoutingEvent` with source `escalation`, the
signal and score. In the `model_routing` block, `decision_source` records the
last step that changed the model, and `escalation_attempted` is kept next to
`fallback_attempted`, with the discarded attempt's tokens. Today
`fallback_attempted` is derived from the final model differing from the
selected one; with escalation it has to come from an actual fallback attempt.
Two metrics next to `routingDecisionLatencyMs`: `routingEscalationCount` and
`routingEscalatedTokens`. An escalated request pays the cheap call, the strong
call and any judge or decision call, so the cascade costs more than calling the
strong model directly once the escalation rate exceeds roughly (big minus small
minus judge) over big. The docs should show that, and #1065, the per-agent cost
budget, is where a ceiling belongs.
**Scope:** Java first. Python follows the parity work in #1062; registering a
`MODEL_ROUTER` from Python raises today and the routing event is Java-only.
## Questions
1. Separate `escalation` policy with its own tiers, or widen `fallback` with a
quality trigger and reuse declaration order? I lean separate, so the error path
does not change.
2. Should answer confidence ship built in, or only as a custom signal with an
example? I lean built in, with the calibration caveat in the docs.
3. Sticky escalation for the whole request, or per-round inside the tool loop?
4. Should `strategy(...)` become optional when tiers are set?
5. For answer confidence, is adding a field to the caller's structured-output
schema acceptable, or should it be a separate signal call?
6. When the last tier fails, fail the request, or return the discarded cheaper
answer flagged `escalation_failed`? I lean fail, so callers never get an answer
the policy rejected.
References: [OpenRouter: confidence
thresholds](https://openrouter.ai/blog/insights/confidence-thresholds-for-model-escalation-routing/),
[OpenRouter: Jev-verified
cascade](https://openrouter.ai/docs/guides/community/jev), [Vals AI: Jev
evaluation](https://www.vals.ai/blogs/independent-evaluation-of-jev), [Bedrock
Intelligent Prompt
Routing](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-routing.html),
[LangChain4j model
router](https://docs.langchain4j.dev/integrations/model-router/), [Portkey
guardrails](https://portkey.ai/docs/product/guardrails), [NVIDIA NeMo
Switchyard escalation
router](https://github.com/NVIDIA-NeMo/Switchyard/blob/main/docs/routing_algorithms/escalation_router_routing.md),
[FrugalGPT](https://arxiv.org/abs/2305.05176),
[AutoMix](https://arxiv.org/abs/2310.12963), [Bouchard, Is escalation worth
it?](https://arxiv.org/abs/2605.06350), [Gupta et al., LM
cascades](https://arxiv.org/abs/2404.10136).
GitHub link: https://github.com/apache/flink-agents/discussions/1328
----
This is an automatically sent email for [email protected].
To unsubscribe, please send an email to: [email protected]
