purushah opened a new issue, #1325: URL: https://github.com/apache/flink-agents/issues/1325
### Search before asking - [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar. ### Description Thanks @joeyutong for adding decision model support in #1264. It fits the routing use case well. I'd like to add a new built-in routing strategy alongside `Strategies.rules(...)`, `Strategies.llm(...)` and `Strategies.custom(...)`: one backed by a `DECISION_MODEL` resource (#1243, #1264). **Why.** The LLM judge returns a candidate name and nothing else. Decision models return a choice plus a probability per option, with no verdict parsing; a one-line instruction such as "Which model should answer this request?" replaces the judge prompt template. That gives routing two things it does not have today: a confidence for the decision, and a way to abstain when the confidence is low. **Prior art.** Two frameworks already do this. LangChain4j has `RoutingChatModel` with a `DecisionModelChatModelRouter`: routes declared by name and description, a mandatory default route, `minProbability(0.7)` sends uncertain decisions to the default, and the rounds of a tool-calling loop stay on the chosen model. LangChain's Python `OpenAIModelRouterMiddleware` classifies the latest human message across named models, uses the choice for the whole run, and stores choice, probabilities and confidence in agent state, with no threshold. The shape below is the same idea on our router, where the default model, sticky tool rounds and durable replay already exist. **Proposed API.** ```java ModelRouter.of("small", "big") .describe("small", "short factual questions and everyday requests") .describe("big", "code, SQL, math, and multi-step reasoning") .strategy(Strategies.decision("dm").withMinProbability(0.6)) .defaultModel("small") .fallback(true) .build(); ``` `dm` is a registered decision model setup. Names are illustrative. **Behaviour.** - The executor builds one `ChoiceQuestion`. Candidate names are the choices, the `describe(...)` texts are the criteria, the conversation text is the input. Decision input is text and JSON only, so media is not passed. `withMaxContextChars` applies with the same rules as for the judge; today it is restricted to `Strategies.llm`, so that guard is widened. - `ChoiceAnswer.choice` becomes the selected model. Its probability is recorded in the existing `RoutingDecision.score` field and in the `model_routing` block. - `withMinProbability(p)` is optional. The score is the probability of the chosen model as the provider reported it; the framework does not normalize the map, so the executor does not assume the values sum to one. If the score is below `p`, or the answer is a `RefusalAnswer`, the strategy abstains and the default model is used, with the reason and the full probability map in the routing metadata. `provider_confidence`, when the provider reports one, is copied into the metadata but not used for the threshold, since not every provider returns it; a separate knob could follow if the data supports it. - The executor calls the setup directly inside `durableExecute` under the id `decide:<router>`, like the judge under `judge:<router>`, so a recorded verdict is reused on replay. This is distinct from the flat `decide` id of the built-in decision action, which the executor does not go through, so it owns its own retries, execution report and token metrics. Retries follow the configured retry budget. A call that exhausts retries fails the request, same as the judge. - `ModelRoutingEvent` gets a new decision source, `decision_model`. The executor records usage on the decision model's metrics, as the judge does on the chat model's. **Validation at plan time.** The setup exists and is a decision model, every candidate has a description, and `p` is in `[0, 1]`. **Scope.** Java first, since routers are Java only today. The built-in decision providers are Python, so a Java job using one needs the Python environment on the TaskManagers, as for any Java decision call. Python routing follows the parity work in #1062. Touch list: one executor class, a `RoutingStrategyType` value with its `validate` and `RoutingExecutors` entries, the plan-time validation switch, one factory method, tests, a docs section on the Model Routing page. **Not in scope.** Using a decision model to check the answer after the call. That is a separate proposal. Related: #897, #1062, #1243, #1264, #1304. Prior art: [LangChain4j model routing](https://docs.langchain4j.dev/tutorials/model-routing), [LangChain OpenAI decisions integration](https://docs.langchain.com/oss/python/integrations/decision_models/openai). ### Are you willing to submit a PR? - [x] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
