GitHub user wenjin272 added a comment to the discussion: [Discussion] Quality-triggered escalation for model routing
Thanks for the proposal. My preferences on the six questions are: 1. **Keep escalation separate from fallback.** They address different scenarios: fallback handles invocation failures, while escalation handles successful responses with insufficient quality. Users should be able to configure them independently. 2. **Start with a custom quality-signal interface and examples.** Built-in answer confidence would require changes to prompts, output schemas, and response parsing. I would defer that until we have more experience with concrete use cases. 3. **Keep escalation sticky for the request.** Once a request moves to a stronger model, subsequent tool rounds should continue using it. 4. **Allow `strategy(...)` to be omitted when tiers are configured**, defaulting to the first tier. 5. **Defer the schema-modification question**, since we would start with custom signals rather than built-in answer confidence. 6. **Fail the request when retries and any applicable fallback options are exhausted**, rather than returning an answer that the quality policy already rejected. I would also appreciate some clarification on two aspects: - **What exactly are we evaluating, and how is “good enough” defined?** Structural validity, a model’s self-reported confidence, and whether an answer is correct or satisfies the request are different signals. What criteria would the evaluator use, and which parts would users need to define? It would help to walk through concrete examples for both a completed answer and a response containing only tool calls, including the evaluator’s inputs, the acceptance criteria, and how the result triggers escalation. - **What happens to prior progress when escalation occurs mid-request?** Does the stronger model receive the existing conversation and tool results, or restart from the original request? Checking before dispatching the current round’s tools prevents side effects from that round, but earlier rounds may already have produced side effects. How would those results be preserved and repeated execution of those operations avoided? GitHub link: https://github.com/apache/flink-agents/discussions/1328#discussioncomment-19261129 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
