GitHub user joeyutong edited a comment on the discussion: [Discussion][Observability 2/2] Built-in Operational Metrics for Flink Agents
Thanks for raising these points. I agree that these contracts should be made more explicit. **Retry scope and compatibility.** The move from `model` to `model_resource` is an intentional breaking change to the metric scope; the compatibility statement applies only to the existing metric names. The scope value identifies the configured ChatModel setup selected by `ChatRequestEvent.model`, not the underlying ChatModel connection. LLM health and retry metrics belong to the setup because they describe the framework-level logical call and retry loop in `ChatModelAction`. Connection or provider SDK transport retries are a separate lower-level signal and are not covered by these metrics. **MCP Server identity.** The Part 1 prototype already propagates this identity in both Java and Python. When an MCP Server declared in the AgentPlan is expanded into Tool resources, each discovered MCP Tool retains the declared server resource name. At execution time, the Tool exposes that name through the common Tool execution metadata contract, and the metrics recorder uses it as the `mcp_server` scope. Both runtime paths use the same metadata key and semantics. If that identity is unavailable, the runtime does not infer it from the endpoint and does not emit the server-scoped metric. **Latency aggregation.** Agreed. The current latency histograms are scoped to an Action and resource, so their exported percentiles should not be treated as generally composable across Actions. The resource-wide aggregation statement applies to counters such as call successes and failures, not to latency percentiles. The proposal now clarifies that `llmCallLatencyMs`, Tool latency, Skill latency, and MCP latency describe their respective per-Action resource scopes. **Gauge recovery semantics.** In the prototype, all four proposed Gauges are reconstructed from restored runtime state. Active input runs come from the restored processing-key state, pending input Events from the pending-Event state, pending Action tasks from the restored ActionTask queues, and active Action executions from restored tasks whose execution-start marker is set. These Gauges are reconstructed from checkpointed state rather than reset or treated as best-effort after failover. The proposal now spells out these state sources. **Tool success/failure parity.** Tool metrics follow each runtime's existing Tool contract and do not interpret arbitrary return payloads. Java treats an unsuccessful `ToolResponse` as failure. Python treats resource preparation or invocation exceptions as failure, but any normal `Tool.call` return remains successful because Python currently has no equivalent explicit error-result type. The proposal now states this limitation explicitly; strict alignment is tracked in [#956](https://github.com/apache/flink-agents/issues/956) after the parallel Tool-call work in [#926](https://github.com/apache/flink-agents/pull/926). GitHub link: https://github.com/apache/flink-agents/discussions/901#discussioncomment-17698167 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
