GitHub user purushah added a comment to the discussion: [Discussion] Pluggable model routing — API & design (PR #852 follow-up)
Thanks @weiqingy — good catch that this is an interface question, not just an implementation detail. I agree that if `Strategies.llm(...)` shipped under a plain synchronous `route()`, the judge call would likely sit inside `route()` as a hidden `judge.chat(...)`. Moving that later onto a framework-managed observable path could require changing the extension point, not just changing the implementation. The direction I want for observable LLM routing is: the framework runs the judge, not the strategy. Concretely, LLM routing should be expressed as a framework-managed routing step: the routing code identifies that a judge model needs to be invoked, and `ChatModelAction` executes that judge call through the normal durable/metered/observable chat path before continuing with the selected backend model. That keeps the judge from being a black box, and keeps the durable execution and observability hooks in the framework where they already live. Given that, I think `Strategies.llm(...)` should be an explicit follow-up rather than part of the first implementation PR. It pulls in the two-phase routing shape, judge orchestration inside `ChatModelAction`, and a second model call with its own durability/observability semantics. Folding that into v1 would make the PR larger and harder to review. So for v1, I'll keep the scope to framework-owned routing in `ChatModelAction`, structured `RoutingDecision`, observability-only `ModelRoutingEvent`, route-once-per-`initialRequestId` behavior across tool rounds, fallback semantics, and rule/custom pure-function strategies through the simple `route()` contract. `Strategies.llm(...)` will be documented as a follow-up, implemented through the framework-managed observable path rather than as a hidden synchronous model call inside `route()`. Agreed that the rest are implementation-contract details for the PR. Happy to close this discussion and continue there. Thanks again for the careful review. GitHub link: https://github.com/apache/flink-agents/discussions/897#discussioncomment-17802481 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
