GitHub user joeyutong created a discussion: [Discussion][Observability 1/2] Recording Agent Traces in the Event Log
This proposal is the first in a two-part Flink Agents observability series: 1. **Recording Agent Traces in the Event Log** (this proposal) 2. **Built-in Operational Metrics for Flink Agents** (follow-up) The two proposals serve complementary purposes. Agent Trace reconstructs what happened during a single run, while operational metrics describe the aggregate health and behavior of many runs. ## 1. Context and Existing Work Flink Agents currently provides three main observability surfaces: metrics, Event Log, and `EventListener`. They provide aggregate signals, persisted business Event records, and callback hooks, but do not capture enough context to reconstruct an agent run as a causal trace. This proposal builds on the gap identified in [Discussion #710](https://github.com/apache/flink-agents/discussions/710) and focuses on recording. Related community work includes: | Community work | Existing progress | Relationship to this proposal | | --- | --- | --- | | [Discussion #710](https://github.com/apache/flink-agents/discussions/710) | Identifies the lack of per-run causal traces and separates recording from reconstruction and visualization. | Provides the overall problem statement; this proposal addresses the recording side. | | [Issue #841](https://github.com/apache/flink-agents/issues/841) | Proposes adding run identity, the emitting Action, and source Event information to existing Event Log records. | Overlaps with this proposal on business Event lineage. This proposal also introduces Execution Events for Action, LLM, Parser, and Tool execution boundaries. | | [Discussion #876](https://github.com/apache/flink-agents/discussions/876) and [PR #887](https://github.com/apache/flink-agents/pull/887) | Extend the business Event taxonomy with typed Memory Events and an `AgentRunBeginEvent` carrying a short-term-memory snapshot. | This proposal retains them as business Events and applies the same trace context to them as to other logged Events. | To close the remaining recording gap, this proposal combines two complementary forms of context: ```text Agent Trace = business Event causality + execution hierarchy ``` ## 2. Motivating Example Consider an Action that calls an LLM and parses the response into a structured output: ```text InputEvent -> chat_model_action -> LLM call: success -> structured-output parser: failed ``` The Event Log may contain the original input, a chat request, a chat response, and an Action failure. Reconstructing the complete run requires both lineage and execution information. Run identity and business Event lineage answer: - Which records belong to the same run? - Which Event triggered an Action, and which Action emitted a later Event? Execution information answers: - Was the failure caused by the model call or by parsing its response? - Which concrete invocation of `chat_model_action` contained those operations? - How long did the LLM call and Parser execution take independently? Event lineage alone cannot represent the LLM and Parser boundaries because those operations are not business Events. A complete trace therefore needs both forms of context. ## 3. Goals and Non-goals ### Goals - Associate Event Log records with one run, business key, and Agent. - Reconstruct the actual Event-to-Action causal relationship at runtime. - Represent Action, LLM, Parser, and Tool execution boundaries. - Reconstruct parent-child relationships between nested executions. - Record execution outcomes and stable failure categories. - Preserve the original business Event payload. - Keep Java and Python Event Log records semantically aligned. - Extend the existing Event Log instead of introducing an independent tracing backend. ### Non-goals - Trace query APIs or storage indexing. - Trace reconstruction or visualization. - OpenTelemetry export. - Evaluation datasets or experiment tracking. - Dashboards, alerts, or other product-level capabilities. - Provider-attempt-level tracing for retries. - Session-level tracing across multiple inputs. ## 4. Proposed Trace Model The existing Event Log model records a business Event together with its occurrence context: ```text Existing EventLogRecord = Event + EventContext ``` This model records which business Event occurred and when it was processed. It does not identify the run or execution to which the record belongs, and it cannot represent runtime work that is not a business Event. The proposed model extends the record as follows: ```text Proposed EventLogRecord = Event or Execution Event + EventContext + ExecutionTraceContext ``` The proposal leaves the responsibilities of `Event` and `EventContext` unchanged. It adds Execution Events for runtime lifecycle boundaries and `ExecutionTraceContext` for run identity, Event-to-Action causality, and execution hierarchy. Together, business Events describe workflow causality, while Execution Events describe the runtime work performed at each Action node. | Abstraction | Responsibility | | --- | --- | | `Event` | Describes what happened through an Event id, type, and attributes. | | Execution Event | Describes the lifecycle and result of one execution. | | `EventContext` | Carries occurrence metadata, currently the Event type and timestamp. | | `ExecutionTraceContext` | Associates the record with a run, Action scope, execution hierarchy, and execution entity. | | `EventLogRecord` | Combines an Event, its occurrence context, and its trace context into one persistent record. | ### 4.1 Business Event and Execution Event Business Events and Execution Events have distinct semantics: - A **business Event** represents a workflow occurrence and retains its original payload. It continues to participate in Event routing, `EventListener` callbacks, and existing business Event metrics. - An **Execution Event** represents the lifecycle of one execution. It records `started`, `finished`, `failed`, or `reused` and is used only for runtime observation. An Execution Event is a runtime recording abstraction, not a new public Event base class. It reuses the existing Event data structure, but does not enter `EventRouter`, notify `EventListener`, trigger Actions, or contribute to business Event metrics. Trace information remains outside user-owned Event attributes. A typed business Event, including a Memory Event or `AgentRunBeginEvent`, can receive the same run and execution context when written without changing its payload schema. ### 4.2 Trace Relationships In this proposal, a **run** is one processing instance initiated by an input for one business key. It is not a user session and does not span multiple inputs. The examples use `input_run_id` to make this scope explicit; the final name can be aligned with the existing Agent Run terminology. An **execution** is one concrete invocation of an Action, LLM, Parser, or Tool. Invoking the same entity twice within one run creates two distinct executions. The trace combines two relationships: 1. Business Events connect Action executions into a causal graph. 2. Parent execution identifiers connect operations performed inside an Action into an execution tree. ```mermaid flowchart TB IN["Input Event"] -->|"triggers"| A1["Action execution A"] A1 -->|"contains"| L1["LLM execution"] A1 -->|"contains"| P1["Parser execution"] A1 -->|"emits"| E1["Business Event E"] E1 -->|"triggers"| A2["Action execution B"] A2 -->|"contains"| T1["Tool execution 1"] A2 -->|"contains"| T2["Tool execution 2"] ``` For business Event causality: - An Action execution records the business Event occurrence that triggered it. - Action executions connected by business Events are siblings rather than parent and child executions. - Static Action trigger rules describe possible routes, but cannot identify the Event occurrence that triggered a concrete Action execution under concurrency, retry, or replay. For execution hierarchy: - The run groups all trace records and serves as the trace root; it is not itself an execution. - LLM, Parser, and Tool executions are children of the Action execution in which they run. Together, business Event records reconstruct the workflow graph, while Execution Events and their parent identifiers reconstruct the work performed within each Action execution. ## 5. Event Log Record The serialized representation remains a flat JSONL record so that commonly queried fields can be indexed and aggregated without parsing nested envelopes. The in-memory model preserves the separation between `Event`, `EventContext`, and `ExecutionTraceContext`. | Field group | Fields | Source | Purpose | | --- | --- | --- | --- | | Event occurrence context | `timestamp` | `EventContext` | Records when the Event occurrence was processed. | | Event | `event_id`, `event_type`, `event_attributes` | `Event` or Execution Event | Preserve the Event identity, semantics, and payload. | | Run identity | `input_run_id`, `business_key`, `agent_name` | `ExecutionTraceContext` | Group records from one input-processing run. | | Execution hierarchy | `execution_id`, `parent_execution_id`, `action_trigger_event_id` | `ExecutionTraceContext` | Reconstruct nested executions and Event-to-Action causality. | | Execution entity | `entity_type`, `entity_name`, `entity_metadata` | `ExecutionTraceContext` | Identify the primary execution entity and carry its supplemental metadata. | | Execution outcome | `status`, `problem_category` | Execution Event attributes | Describe the lifecycle result and stable failure category of an Execution Event. | `EventContext` mirrors `Event.getType()` as part of its occurrence contract, and the serializer writes that value once as `event_type`. For Execution Events, `status` and `problem_category` are promoted from Event attributes to top-level fields during serialization. `entity_type` and `entity_name` identify the primary execution node, such as an Action, LLM, Parser, or Tool invocation. `entity_metadata` contains supplemental attributes for that node; it neither introduces another execution node nor defines a parent-child relationship. `parent_execution_id` and `action_trigger_event_id` capture different relationships: - `parent_execution_id` represents execution containment. - `action_trigger_event_id` identifies the business Event occurrence that triggered the current Action scope. Child LLM, Parser, and Tool records may copy the Action's trigger Event id so that the Action scope can be queried directly. This does not imply that the business Event directly invoked the child execution. For example, a structured-output parsing failure may be written as: ```json { "timestamp": "2026-07-02T01:23:45.678Z", "input_run_id": "run-1", "business_key": "order-1", "agent_name": "order-agent", "execution_id": "parser-execution-1", "parent_execution_id": "action-execution-1", "action_trigger_event_id": "input-event-1", "entity_type": "parser", "entity_name": "structured_output", "event_id": "event-17", "event_type": "_execution_failed_event", "status": "failed", "problem_category": "model_output_parse_error", "event_attributes": { "error_type": "JsonProcessingException", "error_message": "Unable to parse the model response" } } ``` `event_id` identifies a logical Event occurrence; it does not imply that the corresponding physical log record is written exactly once. ## 6. Recording Boundaries ### Run Context When an input enters the Agent runtime, the framework creates or restores a run context containing: - `input_run_id` - `business_key` - `agent_name` The same context is propagated to business Events and executions created during the run. ### Action The runtime records one Action execution for each concrete invocation: - `started` before invoking the Action. - `finished` after successful completion. - `failed` when an unhandled Action exception terminates the invocation. - `reused` when `ActionStateStore` returns an already completed Action result without re-executing the Action. Action-level failures use `action_execution_failed`; the runtime does not infer a more specific category from a child failure. LLM, Parser, and Tool executions record their own precise failure categories. ### LLM An LLM execution represents one logical `ChatModel` call rather than an individual provider attempt. In the initial scope, intermediate retry attempts are not separate executions. - A final model response records success. - A final model-call failure records `model_call_failed`. ### Parser Structured-output parsing has its own Parser execution so that consumers can distinguish between: - A model call that failed. - A model call that succeeded but returned an unparseable response. Parser failures use `model_output_parse_error`. Recording the Parser boundary must not change the existing retry or ignore behavior of structured-output processing. ### Tool A Tool execution represents one concrete tool invocation. If a `ToolRequestEvent` requests multiple calls, each invocation produces a separate Tool execution. - A successful Tool response records success. - A Tool error response keeps the original `ToolResponseEvent` payload but records the Tool execution as failed. - A thrown Tool exception records `tool_call_failed` and preserves the underlying error type and message. Tool calls, MCP Servers, and Skills have different roles in the trace model: | Concept | Execution node | Trace representation | | --- | --- | --- | | Tool call | Yes | Each invocation is one Tool execution. | | MCP Tool call | Yes, as a Tool execution | The MCP Server is the resource exposing the Tool, not another execution level. | | MCP Server | No | Recorded as resource provenance on its MCP Tool executions. | | Skill | No | Represents instructions and resources; only an explicit `load_skill` invocation is an execution. | For example, an MCP Tool call has one execution node: ```text Action execution -> Tool execution: search metadata: - tool type: MCP - MCP Server: search_server ``` In the serialized record, `entity_type` and `entity_name` identify the Tool execution as the primary entity. Its Tool kind and MCP Server provenance are stored in `entity_metadata`, not represented as child execution nodes: ```json { "entity_type": "tool", "entity_name": "search", "entity_metadata": { "tool_type": "mcp", "mcp_server": "search_server" } } ``` The MCP Server therefore has no separate execution id or lifecycle Events. Its metadata can still be used to query or aggregate related MCP Tool executions. `load_skill` is a built-in Tool that reads the requested Skill's `SKILL.md` content or a specific resource and returns it to the model. Its invocation is represented as a Tool execution, with the Skill name and optional resource path stored as metadata. The runtime does not propagate a Skill execution context to subsequent Tool calls, so the trace does not attribute those calls to the loaded Skill. ### Retry and Replay Retry and replay do not have dedicated Event types. Depending on the runtime path, a repeated operation appears as a new execution or another physical Event Log record. ## 7. Runtime Integration and Compatibility Business Events and Execution Events follow separate semantic paths but share the same Event Log writer: ```mermaid flowchart TD BE["Business Event"] --> ER["EventRouter"] ER --> EL["EventListener"] ER --> BM["Business Event metrics"] ER --> AR["Action routing"] ER --> W["Event Log writer"] EE["Execution Event"] --> ES["ExecutionEventSink"] ES --> W W --> LOG["Event Log"] ``` This separation preserves existing externally observable behavior: - Business Events retain their existing `EventRouter` behavior. - Execution Events do not trigger Actions or notify user `EventListener` implementations. - Both paths use the same writer lifecycle and output format. - Event Log writes remain best-effort and do not affect the outcome of an Action. - Existing `EventLogger` implementations can continue to handle `Event` and `EventContext` without consuming trace context. New records use a normalized, flat JSON format with snake_case field names. The deserializer continues to support the previous nested format. Legacy records have no run or execution context and therefore cannot form a complete trace. Explicit execution boundaries add lifecycle records, so log volume grows with the number of recorded Action, LLM, Parser, and Tool executions. The proposal uses the existing Event Log path instead of introducing a separate tracing exporter or recording pipeline. ## 8. Recovery and Replay Semantics The Event Log is best-effort and does not provide exactly-once delivery. Without `ActionStateStore`: - Source replay may process the same logical business input again. - The replayed input currently receives new run and execution identities. - Each resulting run still has an internally consistent execution hierarchy. With `ActionStateStore`: - A completed Action can be reused without re-executing user code. - The current run records a `reused` Execution Event for that Action. - Persisted output Events continue through the current run. - The original `ExecutionTraceContext` is not stored with the completed Action, so the Event Log cannot link a replayed output Event to the execution that originally produced it. Consumers must not infer missing identities from timestamps or payload content. ## 9. Prototype Status The model has been prototyped across the Java and Python runtime paths. The prototype validates: - A shared Event Log schema for business Events and Execution Events. - Action, LLM, Parser, and Tool lifecycle records, including success and failure paths. - Event-to-Action causality, nested executions, and multiple Tool calls from one request. - MCP Server and explicit Skill metadata. - Java and Python semantic alignment through conformance tests and end-to-end probes. The purpose of this discussion is to review the model and contracts before submitting the implementation upstream. ## 10. Summary Business Event lineage is necessary for Agent Trace, but it is not sufficient. A complete trace must also represent runtime executions that are not business Events. This proposal extends the existing Event Log with: - Run identity and business key association. - Explicit Event-to-Action causality. - Action, LLM, Parser, and Tool execution lifecycle records. - Parent-child execution relationships. - Stable execution outcomes and failure categories. - A shared recording format for Java and Python. The result is a framework-native recording model that allows consumers to reconstruct a run without changing business Event payloads or introducing a separate tracing system. GitHub link: https://github.com/apache/flink-agents/discussions/900 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
