GitHub user purushah added a comment to the discussion: [Discussion] Quality-triggered escalation for model routing
Thanks. Agreed on all six. One adjustment on 2: I'd keep the structural checks built in, since they need no prompt or schema change, and ship a custom signal interface with one example that uses a decision model on the answer. Built-in answer confidence and the schema question wait. **What is evaluated.** The evaluator gets the request, the candidate's response (content and tool calls) and the candidate's tool definitions, and returns accept or escalate with a reason and an optional score. Structural checks are always on: empty content with no tool call, a call to a tool the candidate does not have, arguments that fail the tool's schema, the same call repeated with identical arguments. Custom signals and the decision-model example are thresholds the user sets. "Good enough" means every configured signal accepts; the framework does not judge correctness itself. *Completed answer:* tiers can be two or three models, cheapest first. With `small`, `medium`, `big`: `small` returns SQL text, structural passes, the decision model scores 0.55 against 0.7, so `medium` gets the original request. `medium` scores 0.65, still below 0.7, so `big` gets it and, as the last tier, its answer is final. Each hop pays for the discarded attempt, so two tiers is the recommended default and three is for when the middle model catches most escalations. *Tool-call-only response:* structural checks the tool exists for `small` and the arguments validate. If that fails we escalate before dispatching, so the bad call has no side effects. The decision-model signal skips tool-only rounds by default. Tool-only rounds are where escalation has the least to say; the quality signal applies to the answer the model writes after the tools return. **Prior progress.** The stronger model continues the conversation, it does not restart. It gets the original request plus every completed round with tool results; only the rejected round is dropped, and its tools never ran. Earlier tool results stay in the conversation and in the journal under `tool-call:<id>`, so nothing is re-executed. A restart would discard that work and is the path that could repeat side effects. Next from me: a short write-up with the signal interface, the example and these walk-throughs, then the PR. The starting tier can come from #1330. GitHub link: https://github.com/apache/flink-agents/discussions/1328#discussioncomment-19274082 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
