GitHub user Rishankgupta08 created a discussion: Design Proposal: Dynamic Runtime Refresh of MCP Tools and Prompts (#458)
<h2><span>Context</span></h2><p><span>This discussion proposes an approach for implementing the remaining runtime-refresh functionality in #458.</span></p><p><a href="https://github.com/apache/flink-agents/pull/1373"><span>PR #1373</span></a><span> moved MCP tool and prompt discovery from plan construction to operator startup for Java and Python. The remaining gap is that the discovered catalog stays static while the job is running.</span></p><p><span>Relevant existing components include:</span></p><ul><li><p><code><span>ActionExecutionOperator.open()</span></code><span> — initializes runtime resource discovery.</span></p></li><li><p><code><span>JavaMCPResourceDiscovery</span></code><span> and </span><code><span>PythonMCPResourceDiscovery</span></code><span> — discover tools and prompts.</span></p></li><li><p><code><span>ResourceCache</span></code><span> — stores and resolves runtime resources.</span></p></li><li><p><code><span>BaseChatModelSetup</span></code><span> — resolve s configured prompts and tools for model requests.</span></p></li><li><p><code><span>ToolCallAction</span></code><span> — resolves and executes tools returned by the model.</span></p></li></ul><p><span>One important consideration: </span><code><span>BaseChatModelSetup</span></code><span> currently resolves its tool list during </span><code><span>open()</span></code><span> and reuses it for subsequent chat calls. Therefore, updating the resource cache alone would not make newly discovered tools available in later model requests.</span></p><h2><span>1. Refresh strategy comparison</span></h2><p><span>I see two primary approaches.</span></p><h3><span>Option A: Notification-driven refresh</span></h3><p><span>Use MCP list-change notifications to detect changes in the server's advertised tool and prompt catalogs.</span></p><p><span>Relevant protocol notifications:</span></p><ul><li><p><code><span>notifications/tools/list_changed</span></code></p></li><li><p><code><span>notifications/prom pts/list_changed</span></code></p></li></ul><p><span>The server advertises </span><code><span>listChanged</span></code><span> support through its capabilities. The client can listen for supported changes and request the updated catalog.</span></p><p><span>References:</span></p><ul><li><p><a href="https://modelcontextprotocol.io/specification/2025-11-25/server/tools"><span>MCP Tools specification</span></a></p></li><li><p><a href="https://modelcontextprotocol.io/specification/2025-11-25/server/prompts"><span>MCP Prompts specification</span></a></p></li></ul><p><strong><span>Flow:</span></strong></p><ol start="1"><li><p><span>Establish the MCP connection and perform initial discovery.</span></p></li><li><p><span>Register the appropriate change-notification handlers or subscriptions.</span></p></li><li><p><span>When a notification arrives, mark the affected catalog as stale.</span></p></li><li><p><span>Schedule a refresh outside the Flink mailbox thread.</span></p></li><li><p><span>Fet ch the latest tools and prompts.</span></p></li><li><p><span>Validate the complete candidate catalog.</span></p></li><li><p><span>Publish the updated catalog atomically.</span></p></li></ol><p><strong><span>Advantages:</span></strong></p><ul><li><p><span>Low refresh latency.</span></p></li><li><p><span>No repeated list requests when nothing changes.</span></p></li><li><p><span>Natural fit for MCP servers supporting list-change notifications.</span></p></li></ul><p><strong><span>Trade-offs:</span></strong></p><ul><li><p><span>Depends on server and client SDK notification support.</span></p></li><li><p><span>Notifications can be missed during disconnection or reconnection.</span></p></li><li><p><span>Multiple notifications can arrive in quick succession and need coalescing.</span></p></li><li><p><span>Java and Python SDK APIs may differ.</span></p></li></ul><h3><span>Option B: On-demand refresh with optional TTL</span></h3><p><span>Maintain a timestamp for the last successful catalog refresh.</span></p><p><span>Before a model request uses the catalog, check whether the configured TTL has expired. If expired, refresh the catalog before building the next request.</span></p><p><span>A single-flight mechanism should prevent multiple concurrent requests from triggering duplicate refresh operations.</span></p><p><strong><span>Advantages:</span></strong></p><ul><li><p><span>Does not require persistent notification listeners.</span></p></li><li><p><span>Can support servers that do not advertise list-change notifications.</span></p></li><li><p><span>Easier to reason about and test across Java and Python.</span></p></li><li><p><span>Provides a recovery mechanism after disconnection.</span></p></li></ul><p><strong><span>Trade-offs:</span></strong></p><ul><li><p><span>Changes may not become visible until the next request.</span></p></li><li><p><span>TTL introduces a bounded period of stale catalog data.</span></p></li><li><p><span>Frequent refreshes can increase MCP server traffic and request latency.</span></p></li></ul><h2><span>2. Proposed direction: Hybrid refresh mechanism</span></h2><p><span>My initial recommendation is a common refresh abstraction supporting both strategies.</span></p><ul><li><p><span>Notifications should trigger an immediate refresh when supported.</span></p></li><li><p><span>TTL-based refresh should provide reconciliation and fallback.</span></p></li><li><p><span>Servers without notification support should still work through on-demand refresh.</span></p></li><li><p><span>Multiple refresh triggers should be coalesced into one operation.</span></p></li></ul><p><span>The refresh policy should be configurable, rather than making notification support mandatory.</span></p><p><span>The exact default TTL and whether notifications should be enabled in the first implementation can be agreed upon before implementation.</span></p><h2><span>3. Runtime catalog and consistency model</span></h2><p><span>I propose introducing an immutable cat alog snapshot for each runtime resource scope.</span></p><p><span>A snapshot would contain:</span></p><ul><li><p><span>Tool definitions and their schemas.</span></p></li><li><p><span>Prompt definitions and metadata.</span></p></li><li><p><span>The owning MCP server for each discovered resource.</span></p></li><li><p><span>A catalog generation identifier.</span></p></li></ul><p><span>The refresh process should build a candidate snapshot separately from the active snapshot.</span></p><p><span>Before publishing it, validate:</span></p><ol start="1"><li><p><span>Duplicate tool names across MCP servers.</span></p></li><li><p><span>Duplicate prompt names across MCP servers.</span></p></li><li><p><span>Conflicts with statically declared resources.</span></p></li><li><p><span>Reserved tool names beginning with </span><code><span>_subagent_</span></code><span>.</span></p></li><li><p><span>Changes to existing tool schemas and descriptions.</span></p></li></ol><p><span>Tools and prompts should retain separate namespaces, consistent with the existing resource model.</span></p><p><span>If validation or discovery fails, reject the candidate and retain the last known-good snapshot. A partially updated catalog must never become visible.</span></p><p><span>Once validated, publish the new snapshot atomically.</span></p><h3><span>Additions, removals and definition changes</span></h3><ul><li><p><strong><span>Addition:</span></strong><span> Newly discovered tools become available to subsequent model requests, subject to the Agent's existing tool-selection and configuration rules.</span></p></li><li><p><strong><span>Removal:</span></strong><span> Removed tools are excluded from subsequent model requests.</span></p></li><li><p><strong><span>Definition change:</span></strong><span> Updated descriptions and input schemas are used in subsequent requests.</span></p></li><li><p><strong><span>Prompt changes:</span></strong><span> The updated prompt catalog should be used for subsequent pr ompt resolution. A newly discovered prompt should not be automatically selected unless the Agent's configuration or prompt-selection policy requests it.</span></p></li></ul><p><span>An important design question is whether all newly discovered MCP tools should automatically become callable by an Agent, or whether existing explicit tool allow-lists should remain authoritative.</span></p><h2><span>4. In-flight requests and resource lifecycle</span></h2><p><span>A refresh must not invalidate a tool call that was already generated from an earlier catalog.</span></p><p><span>For example:</span></p><ol start="1"><li><p><span>Model request A uses catalog generation 1.</span></p></li><li><p><span>The MCP server removes a tool.</span></p></li><li><p><span>The runtime publishes catalog generation 2.</span></p></li><li><p><span>Model request B uses generation 2.</span></p></li><li><p><span>An already-issued tool call from request A must still be handled consistently with generation 1.</span></p ></li></ol><p><span>I propose retaining old catalog generations while requests >or tool executions still reference them.</span></p><p><span>A request-scoped >snapshot reference or generation identifier could be carried through the >model/tool-call execution flow.</span></p><p><span>When no active execution >references an old generation, its retired MCP resources can be closed >safely.</span></p><p><span>This also avoids closing a server connection while >an in-flight tool call still depends on it.</span></p><h2><span>5. Failure >handling and reconnection</span></h2> Scenario | Proposed behaviour -- | -- Notification received | Mark catalog stale and schedule refresh Multiple notifications | Coalesce into one refresh MCP server temporarily unavailable | Retain last known-good catalog Catalog validation fails | Reject candidate snapshot Duplicate names | Reject conflicting candidate with a clear error Server reconnects | Trigger a full refresh Tool removed during execution | Allow existing execution to finish using its retained generation Repeated refreshes | Retire old resources only after references are released Operator shutdown | Close current and retired MCP resources safely <p><span>Refresh failures should be logged and observable through appropriate metrics. Retries should use bounded backoff and should not block the Flink mailbox thread.</span></p><h2><span>6. Proposed Java/Python scope</span></h2><p><span>Both Java and Python runtime paths should follow the same behavioral contract.</span></p><p><span>However, the notification and connection-management implementations should remain language-specific adapters because their MCP SDK APIs and execution models differ.</span></p><p><strong><span>Java path:</span></strong></p><ul><li><p><span>Integrate notification handling with the Java MCP client.</span></p></li><li><p><span>Coordinate refresh through the runtime resource cache.</span></p></li><li><p><span>Ensure publication is safe with respect to concurrent resource lookups and model requests.</span></p></li></ul><p><strong><span>Python path:</span></strong></p><ul><li><p><span>Integrate the corresponding notification mechanism where the Python MCP clie nt exposes it.</span></p></li><li><p><span>Preserve the single-threaded mailbox execution contract of the Python resource cache.</span></p></li><li><p><span>Avoid concurrent mutation of the active catalog during tool execution.</span></p></li></ul><p><span>Before committing to notification-driven parity, I would verify the exact MCP SDK versions currently pinned by this repository and whether they expose the required client-side notification callbacks. If an SDK does not expose them, TTL-based refresh can provide a compatible fallback.</span></p><p><span>The goal should be identical runtime behaviour for Java and Python, even if their internal refresh implementations differ.</span></p><h2><span>7. Proposed sequence diagram</span></h2><pre><code><span>sequenceDiagram participant MCP as MCP Server participant Client as Java/Python MCP Client participant Refresh as Refresh Coordinator participant Catalog as Runtime Catalog participant Agent as Agent / Chat Model participant Tool as ToolCallAction Agent->>Catalog: Read active catalog Catalog-->>Agent: Snapshot generation 1 Agent->>MCP: Initial tools/list and prompts/list MCP-->>Agent: Initial definitions Agent->>Catalog: Publish initial snapshot MCP-->>Client: List-changed notification Client->>Refresh: Mark catalog stale Refresh->>MCP: Fetch latest tools and prompts MCP-->>Refresh: Updated definitions Refresh->>Catalog: Validate candidate snapshot Catalog->>Catalog: Check names and conflicts Catalog-->>Refresh: Validation successful Refresh->>Catalog: Atomically publish generation 2 Agent->>Catalog: Next model request Catalog-->>Agent: Snapshot generation 2 Agent->>Agent: Send updated tool definitions to model Agent->>Tool: Execute returned tool call Tool->>Catalog: Resolve using request generation Catalog-->>Tool: Matching resource Tool->>MCP: Invoke tool MCP-->>Tool: Tool result Catalog->>Catalog: Retire generation 1 when no longer referenced</span></code></pre><p><span>TTL-based refresh would enter the same refresh coordinator at the next model-request boundary when the active snapshot has expired.</span></p><h2><span>8. Testing strategy</span></h2><p><span>I would cover the following contracts:</span></p><ul><li><p><span>Initial startup discovery remains unchanged.</span></p></li><li><p><span>Newly added tools and prompts become visible after refresh.</span></p></li><li><p><span>Removed resources disappear from subsequent model requests.</span></p></li><li><p><span>Changes to existing tool schemas and descriptions are reflected.</span></p></li><li><p><span>Repeated notifications are coalesced.</span></p></li><li><p><span>TTL expiration triggers refresh.</span></p></li><li><p><span>Servers without notification support still work.</span></p></li><li><p><span>Duplicate names and reserved tool names reject invalid snapshots.</span></p></li><li><p><sp an>Failed refreshes preserve the last known-good snapshot.</span></p></li><li><p><span>Reconnection triggers rediscovery.</span></p></li><li><p><span>In-flight tool executions can complete across catalog generations.</span></p></li><li><p><span>Retired MCP resources are closed only after active references are released.</span></p></li><li><p><span>Repeated refreshes and operator shutdown do not leak resources.</span></p></li><li><p><span>Java and Python implementations follow the same behavioural contracts.</span></p></li></ul><h2><span>9. Questions for maintainer feedback</span></h2><p><span>Before implementation, I would appreciate feedback on these points:</span></p><ol start="1"><li><p><span>Would you prefer notification-driven refresh as the primary strategy, on-demand TTL refresh, or the hybrid approach described above?</span></p></li><li><p><span>Should the first implementation cover both Java and Python, or should we establish the Java path first and add Python parity in a fo llow-up?</span></p></li><li><p><span>Should newly discovered MCP tools automatically be exposed to an Agent, or should the existing explicit tool-selection configuration remain authoritative?</span></p></li><li><p><span>Should the initial implementation refresh only the tools and prompts exposed by already configured MCP servers, leaving dynamic addition/removal of server definitions out of scope?</span></p></li><li><p><span>What would be an appropriate default TTL and failure policy?</span></p></li></ol><p><span>I would like to agree on these behavioural contracts and the initial scope before making implementation changes.</span></p><p><span>Thanks for the guidance! Looking forward to discussing the trade-offs and refining the design with the maintainers.</span></p>## Context This discussion proposes an approach for implementing the remaining runtime-refresh functionality in #458. [[PR #1373](https://github.com/apache/flink-agents/pull/1373)](https://github.com/apache/flink-agents/pull/1373) moved MCP tool and prompt discovery from plan construction to operator startup for Java and Python. The remaining gap is that the discovered catalog stays static while the job is running. Relevant existing components include: - `ActionExecutionOperator.open()` — initializes runtime resource discovery. - `JavaMCPResourceDiscovery` and `PythonMCPResourceDiscovery` — discover tools and prompts. - `ResourceCache` — stores and resolves runtime resources. - `BaseChatModelSetup` — resolves configured prompts and tools for model requests. - `ToolCallAction` — resolves and executes tools returned by the model. One important consideration: `BaseChatModelSetup` currently resolves its tool list during `open()` and reuses it for subsequent chat calls. Therefore, updating the resource cache alone would not make newly discovered tools available in later model requests. ## 1. Refresh strategy comparison I see two primary approaches. ### Option A: Notification-driven refresh Use MCP list-change notifications to detect changes in the server's advertised tool and prompt catalogs. Relevant protocol notifications: - `notifications/tools/list_changed` - `notifications/prompts/list_changed` The server advertises `listChanged` support through its capabilities. The client can listen for supported changes and request the updated catalog. References: - [[MCP Tools specification](https://modelcontextprotocol.io/specification/2025-11-25/server/tools)](https://modelcontextprotocol.io/specification/2025-11-25/server/tools) - [[MCP Prompts specification](https://modelcontextprotocol.io/specification/2025-11-25/server/prompts)](https://modelcontextprotocol.io/specification/2025-11-25/server/prompts) **Flow:** 1. Establish the MCP connection and perform initial discovery. 2. Register the appropriate change-notification handlers or subscriptions. 3. When a notification arrives, mark the affected catalog as stale. 4. Schedule a refresh outside the Flink mailbox thread. 5. Fetch the latest tools and prompts. 6. Validate the complete candidate catalog. 7. Publish the updated catalog atomically. **Advantages:** - Low refresh latency. - No repeated list requests when nothing changes. - Natural fit for MCP servers supporting list-change notifications. **Trade-offs:** - Depends on server and client SDK notification support. - Notifications can be missed during disconnection or reconnection. - Multiple notifications can arrive in quick succession and need coalescing. - Java and Python SDK APIs may differ. ### Option B: On-demand refresh with optional TTL Maintain a timestamp for the last successful catalog refresh. Before a model request uses the catalog, check whether the configured TTL has expired. If expired, refresh the catalog before building the next request. A single-flight mechanism should prevent multiple concurrent requests from triggering duplicate refresh operations. **Advantages:** - Does not require persistent notification listeners. - Can support servers that do not advertise list-change notifications. - Easier to reason about and test across Java and Python. - Provides a recovery mechanism after disconnection. **Trade-offs:** - Changes may not become visible until the next request. - TTL introduces a bounded period of stale catalog data. - Frequent refreshes can increase MCP server traffic and request latency. ## 2. Proposed direction: Hybrid refresh mechanism My initial recommendation is a common refresh abstraction supporting both strategies. - Notifications should trigger an immediate refresh when supported. - TTL-based refresh should provide reconciliation and fallback. - Servers without notification support should still work through on-demand refresh. - Multiple refresh triggers should be coalesced into one operation. The refresh policy should be configurable, rather than making notification support mandatory. The exact default TTL and whether notifications should be enabled in the first implementation can be agreed upon before implementation. ## 3. Runtime catalog and consistency model I propose introducing an immutable catalog snapshot for each runtime resource scope. A snapshot would contain: - Tool definitions and their schemas. - Prompt definitions and metadata. - The owning MCP server for each discovered resource. - A catalog generation identifier. The refresh process should build a candidate snapshot separately from the active snapshot. Before publishing it, validate: 1. Duplicate tool names across MCP servers. 2. Duplicate prompt names across MCP servers. 3. Conflicts with statically declared resources. 4. Reserved tool names beginning with `_subagent_`. 5. Changes to existing tool schemas and descriptions. Tools and prompts should retain separate namespaces, consistent with the existing resource model. If validation or discovery fails, reject the candidate and retain the last known-good snapshot. A partially updated catalog must never become visible. Once validated, publish the new snapshot atomically. ### Additions, removals and definition changes - **Addition:** Newly discovered tools become available to subsequent model requests, subject to the Agent's existing tool-selection and configuration rules. - **Removal:** Removed tools are excluded from subsequent model requests. - **Definition change:** Updated descriptions and input schemas are used in subsequent requests. - **Prompt changes:** The updated prompt catalog should be used for subsequent prompt resolution. A newly discovered prompt should not be automatically selected unless the Agent's configuration or prompt-selection policy requests it. An important design question is whether all newly discovered MCP tools should automatically become callable by an Agent, or whether existing explicit tool allow-lists should remain authoritative. ## 4. In-flight requests and resource lifecycle A refresh must not invalidate a tool call that was already generated from an earlier catalog. For example: 1. Model request A uses catalog generation 1. 2. The MCP server removes a tool. 3. The runtime publishes catalog generation 2. 4. Model request B uses generation 2. 5. An already-issued tool call from request A must still be handled consistently with generation 1. I propose retaining old catalog generations while requests or tool executions still reference them. A request-scoped snapshot reference or generation identifier could be carried through the model/tool-call execution flow. When no active execution references an old generation, its retired MCP resources can be closed safely. This also avoids closing a server connection while an in-flight tool call still depends on it. ## 5. Failure handling and reconnection | Scenario | Proposed behaviour | | --- | --- | | Notification received | Mark catalog stale and schedule refresh | | Multiple notifications | Coalesce into one refresh | | MCP server temporarily unavailable | Retain last known-good catalog | | Catalog validation fails | Reject candidate snapshot | | Duplicate names | Reject conflicting candidate with a clear error | | Server reconnects | Trigger a full refresh | | Tool removed during execution | Allow existing execution to finish using its retained generation | | Repeated refreshes | Retire old resources only after references are released | | Operator shutdown | Close current and retired MCP resources safely | Refresh failures should be logged and observable through appropriate metrics. Retries should use bounded backoff and should not block the Flink mailbox thread. ## 6. Proposed Java/Python scope Both Java and Python runtime paths should follow the same behavioral contract. However, the notification and connection-management implementations should remain language-specific adapters because their MCP SDK APIs and execution models differ. **Java path:** - Integrate notification handling with the Java MCP client. - Coordinate refresh through the runtime resource cache. - Ensure publication is safe with respect to concurrent resource lookups and model requests. **Python path:** - Integrate the corresponding notification mechanism where the Python MCP client exposes it. - Preserve the single-threaded mailbox execution contract of the Python resource cache. - Avoid concurrent mutation of the active catalog during tool execution. Before committing to notification-driven parity, I would verify the exact MCP SDK versions currently pinned by this repository and whether they expose the required client-side notification callbacks. If an SDK does not expose them, TTL-based refresh can provide a compatible fallback. The goal should be identical runtime behaviour for Java and Python, even if their internal refresh implementations differ. ## 7. Proposed sequence diagram ```mermaid sequenceDiagram participant MCP as MCP Server participant Client as Java/Python MCP Client participant Refresh as Refresh Coordinator participant Catalog as Runtime Catalog participant Agent as Agent / Chat Model participant Tool as ToolCallAction Agent->>Catalog: Read active catalog Catalog-->>Agent: Snapshot generation 1 Agent->>MCP: Initial tools/list and prompts/list MCP-->>Agent: Initial definitions Agent->>Catalog: Publish initial snapshot MCP-->>Client: List-changed notification Client->>Refresh: Mark catalog stale Refresh->>MCP: Fetch latest tools and prompts MCP-->>Refresh: Updated definitions Refresh->>Catalog: Validate candidate snapshot Catalog->>Catalog: Check names and conflicts Catalog-->>Refresh: Validation successful Refresh->>Catalog: Atomically publish generation 2 Agent->>Catalog: Next model request Catalog-->>Agent: Snapshot generation 2 Agent->>Agent: Send updated tool definitions to model Agent->>Tool: Execute returned tool call Tool->>Catalog: Resolve using request generation Catalog-->>Tool: Matching resource Tool->>MCP: Invoke tool MCP-->>Tool: Tool result Catalog->>Catalog: Retire generation 1 when no longer referenced ``` TTL-based refresh would enter the same refresh coordinator at the next model-request boundary when the active snapshot has expired. ## 8. Testing strategy I would cover the following contracts: - Initial startup discovery remains unchanged. - Newly added tools and prompts become visible after refresh. - Removed resources disappear from subsequent model requests. - Changes to existing tool schemas and descriptions are reflected. - Repeated notifications are coalesced. - TTL expiration triggers refresh. - Servers without notification support still work. - Duplicate names and reserved tool names reject invalid snapshots. - Failed refreshes preserve the last known-good snapshot. - Reconnection triggers rediscovery. - In-flight tool executions can complete across catalog generations. - Retired MCP resources are closed only after active references are released. - Repeated refreshes and operator shutdown do not leak resources. - Java and Python implementations follow the same behavioural contracts. ## 9. Questions for maintainer feedback Before implementation, I would appreciate feedback on these points: 1. Would you prefer notification-driven refresh as the primary strategy, on-demand TTL refresh, or the hybrid approach described above? 2. Should the first implementation cover both Java and Python, or should we establish the Java path first and add Python parity in a follow-up? 3. Should newly discovered MCP tools automatically be exposed to an Agent, or should the existing explicit tool-selection configuration remain authoritative? 4. Should the initial implementation refresh only the tools and prompts exposed by already configured MCP servers, leaving dynamic addition/removal of server definitions out of scope? 5. What would be an appropriate default TTL and failure policy? I would like to agree on these behavioural contracts and the initial scope before making implementation changes. Thanks for the guidance! Looking forward to discussing the trade-offs and refining the design with the maintainers. GitHub link: https://github.com/apache/flink-agents/discussions/1390 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
