GitHub user weiqingy added a comment to the discussion: [Discussion][Observability 2/2] Built-in Operational Metrics for Flink Agents
Thanks for writing this up. The `model_resource` / provider `model` split in §3.1 looks like an important cleanup to me, especially because retry health and token attribution answer different questions. A few edge cases seem worth clarifying before the metric contract lands. **Retry scope and compatibility.** §3.2 says existing retry names are retained for compatibility, but the proposed scope appears to move from `model=...` to `model_resource=...`. Today the retry value also looks closer to the connection name than the ChatModel setup name. Is the `model` → `model_resource` move intended as a breaking metric-scope change, with compatibility limited to the metric names? And should LLM health/retry metrics be scoped to the configured ChatModel setup, or to the underlying connection where transport failures and retries actually happen? **MCP Server identity.** §4.5 adds `mcp_server` metrics, but it is not clear that the declared MCP Server name is available at Tool execution time in both Java and Python paths. Would Part 1 thread the declared server name through Tool execution metadata, or should the `mcp_server` metric scope be deferred until that identity is available consistently? **Latency aggregation.** §3.1 says consumers can aggregate across the Action scope to view resource-wide health. That works for counters, but latency histograms/percentiles generally do not compose across per-Action series. Is `llmCallLatencyMs` intended to be per-Action latency, or resource-wide latency? Either seems reasonable, but the contract should probably avoid promising resource-wide latency aggregation from sharded histograms. The same question seems to apply to Tool, Skill, and MCP latency histograms. **Gauge recovery semantics.** §6 says Gauges may be reconstructed from restored runtime state when the state identifies pending or active work. Could we spell out which gauges are actually reconstructable after failover? `numOfActiveInputRuns` and `numOfPendingInputEvents` seem to have clearer restored-state sources than `numOfPendingActionTasks`, where the restored action-task queue may not expose an exact size. It would help to distinguish “correctly reconstructed” gauges from “reset or best-effort after task-attempt restart.” **Tool success/failure parity.** §5.4 says a Tool error response is counted as a failed Tool execution, and §7 makes Java/Python outcome alignment part of the contract. Could we define whether Tool success/failure is a framework-level contract or a best-effort interpretation of the Tool return value? Java has `ToolResponse.success/error`, while Python tools currently look more raw-value oriented, so `numOfToolCallsFailed` could mean different things unless the contract is tightened or documented as best-effort. GitHub link: https://github.com/apache/flink-agents/discussions/901#discussioncomment-17693047 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
