weiqingy commented on code in PR #955:
URL: https://github.com/apache/flink-agents/pull/955#discussion_r3911064809
##########
docs/content/docs/operations/monitoring.md:
##########
@@ -36,11 +36,62 @@ We offer data monitoring for built-in metrics, which
includes events, actions, a
| **Agent** | numOfEventProcessedPerSec | The number of
Events this operator has processed per second. | Meter |
| **Agent** | numOfActionsExecuted | The total
number of actions this operator has executed. | Count |
| **Agent** | numOfActionsExecutedPerSec | The number of
actions this operator has executed per second. | Meter |
+| **Agent** | numOfInputRunsSucceeded | The number of
input runs that reached the run-completion boundary. | Count |
+| **Agent** | numOfInputRunsFailed | The number of
input runs terminated by an unhandled exception. | Count |
+| **Agent** | inputRunLatencyMs | End-to-end
input-run latency from entering the agent operator to completion or failure,
including time queued behind another input with the same key. | Histogram |
+| **Agent** | inputRunQueueLatencyMs | Time from
entering the agent operator until the input run starts processing. | Histogram |
+| **Agent** | inputRunProcessingLatencyMs | Time from the
input-run start boundary until completion or failure. | Histogram |
+| **Agent** | numOfPendingInputEvents | Current
number of input Events buffered behind an active run with the same key. | Gauge
|
+| **Agent** | numOfActiveInputRuns | Current
number of logical input runs that are processing or waiting for asynchronous
work. | Gauge |
| **Action** | action.\<action_name\>.numOfActionsExecuted | The total number
of actions this operator has executed for a specific action name. | Count |
| **Action** | action.\<action_name\>.numOfActionsExecutedPerSec | The number
of actions this operator has executed per second for a specific action name. |
Meter |
+| **Action** | action.\<action_name\>.actionSchedulingLatencyMs | Time from
enqueuing the initial Action task until it is selected for execution. |
Histogram |
+| **Action** | action.\<action_name\>.actionExecutionLatencyMs | End-to-end
latency of one logical Action execution, including asynchronous waits and
continuations. | Histogram |
+| **Action** | action.\<action_name\>.numOfPendingActionTasks | Current number
of physical Action task segments waiting to run, including continuations. |
Gauge |
+| **Action** | action.\<action_name\>.numOfActiveActionExecutions | Current
number of logical Action executions that have started but have not reached a
terminal state. | Gauge |
| **Agent** | eventLogTruncatedEvents | Number of
event log records whose payload was truncated at `STANDARD` level. Increments
once per event, regardless of how many fields inside it were truncated. Use
this to decide whether to raise truncation thresholds or move specific event
types to `VERBOSE`. | Count |
| **Agent** | eventLogWriteFailures | Number of
Event Log write attempts for which `append`, `flush`, or both failed. Event Log
writes are best-effort and do not fail the job. | Count |
+For a locally observed input run, `inputRunLatencyMs` is split into queueing
and processing time at the input-run start boundary. `numOfPendingInputEvents`
counts buffered inputs, while `numOfActiveInputRuns` counts logical runs; an
asynchronous run remains active while it is waiting for its continuation.
+
+An Action execution can be active while one of its continuation tasks is
pending, so `numOfActiveActionExecutions` and `numOfPendingActionTasks` are
independent. Action scheduling latency is recorded only for the initial task;
continuation queueing does not create another scheduling sample.
+
+Input-run outcomes and all latency samples are process-local. Runs or Action
executions already in flight when a task is restored do not produce latency
samples because their original timestamps are unavailable. An input Event
restored from the pending queue can still produce an outcome and
processing-latency sample after it starts in the new task attempt, but it does
not produce queue or end-to-end latency. Current-count gauges are rebuilt from
Flink state after restore.
+
+#### Execution Metrics
+
+Execution metrics are derived from LLM and Tool execution lifecycle events.
The `model_resource`, `tool`, `skill`, and `mcp_server` scopes are independent
key-value scopes directly under an Action; none is nested under another. The
existing `model` scope remains dedicated to model usage metrics.
+
+| Scope | Metrics | Description | Type |
+|-------|---------|-------------|------|
+| **Model Resource** |
action.\<action_name\>.model_resource.\<resource_name\>.numOfLlmCallsSucceeded
| The number of framework-observed model invocations that returned
successfully. | Count |
+| **Model Resource** |
action.\<action_name\>.model_resource.\<resource_name\>.numOfLlmCallsFailed |
The number of framework-observed model invocations that failed. | Count |
+| **Model Resource** |
action.\<action_name\>.model_resource.\<resource_name\>.llmCallLatencyMs |
Latency of each framework-observed model invocation, excluding
structured-output parsing and retry wait time. | Histogram |
+| **Model Resource** |
action.\<action_name\>.model_resource.\<resource_name\>.retryCount | The number
of additional model invocations initiated when `ErrorHandlingStrategy.RETRY` is
configured. Only recorded when at least one retry occurs. See
[retry-wait-interval]({{< ref "docs/operations/configuration#core-options"
>}}). | Count |
+| **Model Resource** |
action.\<action_name\>.model_resource.\<resource_name\>.retryWaitSec | The
total backoff time, in seconds, accumulated when `ErrorHandlingStrategy.RETRY`
is configured. Only recorded when at least one retry occurs. | Count |
+| **Tool** | action.\<action_name\>.tool.\<tool_name\>.numOfToolCallsSucceeded
| The number of successful calls to the Tool. | Count |
+| **Tool** | action.\<action_name\>.tool.\<tool_name\>.numOfToolCallsFailed |
The number of failed calls to the Tool. | Count |
+| **Tool** | action.\<action_name\>.tool.\<tool_name\>.toolCallLatencyMs |
Tool call latency. | Histogram |
Review Comment:
`llmCallLatencyMs` at `:69` says exactly what its window covers. This row
says only "Tool call latency", and the window is wider than one tool:
`ToolCallAction.java:136` reports `started` for every tool before any runs, and
the terminals all fire after the batch returns (`:205-208`). So a fast tool in
a batch reports the batch's time, and below JDK 21, where the batch runs
serially (`ContinuationActionExecutor.java:76-95`), it also carries the tools
ahead of it. `skillLoadLatencyMs` and `mcpToolCallLatencyMs` share the value,
and Python matches (`tool_call_action.py:169`, `:219-220`).
The cause sits upstream in `ToolCallAction`, so the doc may be the only
lever in this PR. The `tool.<tool_name>` scope reads like a promise that you
can spot the slow tool in a batch. Is that what you want it to carry, or is the
batch number the one worth publishing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
