purushah commented on code in PR #1124: URL: https://github.com/apache/flink-agents/pull/1124#discussion_r4038916425
########## docs/content/docs/development/model_routing/reference.md: ########## @@ -0,0 +1,51 @@ +--- +title: Reference +weight: 1 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing Reference + +Reference material for [Model Routing]({{< ref "docs/development/model_routing" >}}): event and response fields, metrics, validation stages, and descriptor keys. Review Comment: Done: the Reference subpage is gone and Model Routing is a single page again. The event and response fields and the metrics now sit in Observability, the validation timing is a short note under a small Advanced section, and the descriptor keys are removed. ########## docs/content/docs/development/model_routing/_index.md: ########## @@ -0,0 +1,318 @@ +--- +title: Model Routing +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing + +## Overview + +Model routing lets one chat request choose between several registered chat models at runtime. Instead of naming a chat model in a `ChatRequestEvent`, an agent names a **model router**. The router carries a list of **candidate** chat models and a **routing strategy**. For each request the framework runs the strategy, which either **selects** one candidate or **abstains**, meaning it makes no choice and the router's default model is used. The framework then runs the ordinary chat path against the chosen model. + +The router only selects a model. The chosen model is invoked by the same `ChatModelAction` that serves a plain chat request, so its prompt, tools, skills, retries, token metrics, and event logging apply the same way. A strategy never calls a model itself. When the decision should come from an LLM, the framework runs that **judge** call through its own chat path, see [LLM Judge](#llm-judge). + +Typical uses are sending short requests to a small, cheap model and code, SQL, or multi-step reasoning to a large one; keeping a default model for everything the strategy cannot classify; and falling through to the next candidate when the selected model fails. + +What works with routing, and what does not: + +- Any registered `CHAT_MODEL` from any provider can be a candidate or a judge. Tool calls work: the model that answered the initial request is kept for every tool-call round of that request. +- Routers are declared with `addResource` on the execution environment or on the agent. There is no annotation for declaring a router inside an agent class, and the YAML API has no section for routers. Review Comment: Done for everything that has an annotation: the quick start now declares the connection, both candidates and the judge with `@ChatModelConnection` / `@ChatModelSetup` inside the agent class. The router itself is still registered with `Agent.addResource` in the constructor, and the page says why. I opened #1133 for a `@ModelRouterSetup` annotation (name per your suggestion) so this PR stays documentation-only. ########## docs/content/docs/development/model_routing/_index.md: ########## @@ -0,0 +1,318 @@ +--- +title: Model Routing +weight: 5 +type: docs +--- +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Model Routing + +## Overview + +Model routing lets one chat request choose between several registered chat models at runtime. Instead of naming a chat model in a `ChatRequestEvent`, an agent names a **model router**. The router carries a list of **candidate** chat models and a **routing strategy**. For each request the framework runs the strategy, which either **selects** one candidate or **abstains**, meaning it makes no choice and the router's default model is used. The framework then runs the ordinary chat path against the chosen model. + +The router only selects a model. The chosen model is invoked by the same `ChatModelAction` that serves a plain chat request, so its prompt, tools, skills, retries, token metrics, and event logging apply the same way. A strategy never calls a model itself. When the decision should come from an LLM, the framework runs that **judge** call through its own chat path, see [LLM Judge](#llm-judge). + +Typical uses are sending short requests to a small, cheap model and code, SQL, or multi-step reasoning to a large one; keeping a default model for everything the strategy cannot classify; and falling through to the next candidate when the selected model fails. + +What works with routing, and what does not: + +- Any registered `CHAT_MODEL` from any provider can be a candidate or a judge. Tool calls work: the model that answered the initial request is kept for every tool-call round of that request. +- Routers are declared with `addResource` on the execution environment or on the agent. There is no annotation for declaring a router inside an agent class, and the YAML API has no section for routers. +- `ReActAgent` cannot use a router; it registers its own chat model under a fixed name. Routing is for agents that send `ChatRequestEvent` themselves. +- A router cannot be a judge (rejected when the plan is built), a candidate of another router, or a chat model's `connection` (both fail at request time). + +{{< hint info >}} +Model routing is only supported in Java currently. Python agents cannot register a `MODEL_ROUTER` resource yet and the Python API rejects the attempt with an error. A router declared by a Java agent is still understood when the plan is shared with Python. Python support is planned for a future release. +{{< /hint >}} + +## Declaring a Router + +A router is a resource of type `ResourceType.MODEL_ROUTER`, built with `ModelRouter.of(...)` and registered like any other resource. Its candidates are the names of chat models registered in the same environment, listed in the order [fallback](#default-model-and-fallback) tries them. A name cannot be both a chat model and a router. The routing types live in `org.apache.flink.agents.api.chat.model.routing`. + +{{< tabs "Declaring a Router" >}} + +{{< tab "Java" >}} +```java +import org.apache.flink.agents.api.chat.model.routing.ModelRouter; +import org.apache.flink.agents.api.chat.model.routing.Strategies; +import org.apache.flink.agents.api.resource.ResourceDescriptor; +import org.apache.flink.agents.api.resource.ResourceName; +import org.apache.flink.agents.api.resource.ResourceType; + +import java.util.LinkedHashMap; +import java.util.Map; + +// One Ollama connection shared by the candidates. +agentsEnv.addResource( + "ollamaConnection", + ResourceType.CHAT_MODEL_CONNECTION, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_CONNECTION) + .addInitialArgument("endpoint", "http://localhost:11434";) + .build()); + +// Two candidate chat models registered under the names "small" and "big". +agentsEnv + .addResource( + "small", + ResourceType.CHAT_MODEL, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP) + .addInitialArgument("connection", "ollamaConnection") + .addInitialArgument("model", "qwen3:1.7b") + .build()) + .addResource( + "big", + ResourceType.CHAT_MODEL, + ResourceDescriptor.Builder.newBuilder(ResourceName.ChatModel.OLLAMA_SETUP) + .addInitialArgument("connection", "ollamaConnection") + .addInitialArgument("model", "qwen3:8b") + .build()); + +// Requests whose latest user message mentions code or SQL go to "big"; +// everything else abstains and lands on the default, "small". +Map<String, String> rules = new LinkedHashMap<>(); +rules.put("big", "\\b(code|sql|program|analyze|prove)\\b"); + +agentsEnv.addResource( + "router", + ResourceType.MODEL_ROUTER, + ModelRouter.of("small", "big") + .strategy(Strategies.rules(rules)) + .defaultModel("small") + .fallback(true) + .build()); +``` +{{< /tab >}} + +{{< /tabs >}} + +The agent then names the router in its `ChatRequestEvent`. Nothing else in the agent changes. + +{{< tabs "Using a Router in an Agent" >}} + +{{< tab "Java" >}} +```java +public class ModelRoutingAgent extends Agent { + + /** Send each input to the router, which selects the concrete model. */ + @Action(EventType.InputEvent) + public static void processInput(InputEvent event, RunnerContext ctx) { + ctx.sendEvent( + new ChatRequestEvent( + "router", + Collections.singletonList( + new ChatMessage(MessageRole.USER, (String) event.getInput())))); + } + + /** Emit the model's answer as output. */ + @Action(EventType.ChatResponseEvent) + public static void processChatResponse(ChatResponseEvent event, RunnerContext ctx) { + ctx.sendEvent(new OutputEvent(event.getResponse().getContent())); + } +} +``` +{{< /tab >}} + +{{< /tabs >}} + +| Method | Description | +|--------|-------------| +| `of(String... candidates)` | Start a router over the given chat model names. | +| `strategy(RoutingStrategy)` | Required. One of the `Strategies` factories below. | +| `describe(candidate, description)` | Describe a candidate. Descriptions are the criteria the [LLM judge](#llm-judge) reads. Fails immediately if the name is not a candidate. | +| `defaultModel(String)` | Where the router lands when the strategy abstains. Optional; without it the first candidate is the default. | +| `fallback(boolean)` | Try the remaining candidates, in declaration order, after the selected model fails. Off by default. | +| `build()` | Produces the `ResourceDescriptor` to register. | + +To turn routing off, address a candidate directly in the `ChatRequestEvent`, or replace the router registration with a plain chat model under the same name. + +## Routing Strategies + +A `RoutingStrategy` is a serializable declaration produced by a `Strategies` factory. It travels in the agent plan as a type tag plus arguments; the executor for that type runs on the TaskManager. Every strategy either selects a candidate or abstains. Review Comment: Done: the page is now Overview, Quick Start, Routing Strategies, Default Model and Fallback, Observability, a short Advanced section (validation timing and recovery in two paragraphs), and Examples. Plan type tags, TaskManager executors, durable call ids, the per-provider finish-reason list and the validation stages are gone from the user path. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
