GitHub user joeyutong created a discussion: [Discussion][Observability] Replace Event Log with a Unified Trace Log
This proposal builds on [Recording Agent Traces in the Event Log (#900)](https://github.com/apache/flink-agents/discussions/900). It brings Event and execution logging into a unified Trace Log model while preserving the observation coverage and execution semantics of the existing design. ## 1. Background Event Log currently serves two purposes: recording Events that flow between Actions and recording the execution lifecycle of Actions and their LLM, Parser, and Tool calls. Lifecycle reports are represented as synthetic Events, such as `_execution_finished_event`, even though they are used only for observability and are never routed to Actions. Using Event for both purposes introduces ambiguity into the model and its configuration: - **Event has two meanings.** The same term and data structure describe both objects in the programming model and records used to observe execution. - **Logging configuration spans both models.** Event Log levels and per-type settings control ordinary Event logging, while execution logging requires an additional Trace switch. Selecting execution reports can also require users to know synthetic Event types such as `_execution_finished_event`. - **Execution reports carry redundant metadata.** An execution already has an `executionId` and a `status`, but each lifecycle report also receives an Event UUID and type because it is represented as an Event. We propose **replacing Event Log with a unified Trace Log**. Event will retain its meaning in the programming model: an object that Actions consume or emit. Trace Log will describe Event flow and execution through a common record format and a single set of logging controls, preserving the information currently available in Event Log. ## 2. Goals and Scope ### Goals 1. **Give Event and Trace distinct responsibilities.** Event belongs to the programming model; Trace provides a common representation for observations of Events and execution. 2. **Preserve existing observation coverage.** This includes Event content, execution status, Memory reads and writes, initial Memory snapshots, and the relationships between Events and the executions that produce or consume them. 3. **Unify logging configuration.** Users should be able to choose which records to write, how much content to retain, and where to send the output, with local overrides for specific Event types, Actions, or component calls. ### Scope - The design covers four areas: the data model, runtime collection, configuration, and log consumption. - Event routing, Action execution, and EventListener callbacks retain their existing behavior, as do recovery and result reuse within the same version. - Trace remains best effort. Records may be missing or duplicated; complete execution histories, exactly-once logging, and deduplication after recovery are outside the scope of this proposal. - Migration of persisted runtime state across versions is also outside scope. Upgrade guidance must identify affected checkpoint and savepoint restore paths, as well as reuse of results persisted in ActionStateStore. ## 3. Proposed Design ### 3.1 Data Model and Contract #### Record composition TraceRecord replaces EventLogRecord as the unit written to the log. It contains a TraceContext, an observation timestamp, and attributes, with execution status and failure category where applicable. Event observations and execution reports use this same record type. The following comparison shows the object structures for the same failed execution. Nesting indicates field ownership; the next section illustrates the serialized JSON format. ```text Current: EventLogRecord Proposed: TraceRecord ├── eventContext: EventContext ├── context: TraceContext │ ├── timestamp │ ├── inputRunId, businessKey, agentName │ └── eventType = "_execution_failed_event" │ ├── entityType, entityName ├── traceContext: ExecutionTraceContext │ ├── executionId, parentExecutionId │ ├── inputRunId, businessKey, agentName │ └── entityMetadata │ ├── entityType, entityName ├── timestamp │ ├── executionId, parentExecutionId ├── status = "failed" │ └── entityMetadata ├── problemCategory (when supplied) └── event: Event └── attributes ├── id (generated UUID) ├── errorType ├── type = "_execution_failed_event" └── errorMessage ├── upstreamEventId = null ├── upstreamActionName = null └── attributes ├── status = "failed" ├── problemCategory (when supplied) ├── errorType └── errorMessage ``` Execution identity remains in the context. The synthetic Event, its generated UUID, and its lifecycle type are removed. Status and failure category become fields on TraceRecord, while error details remain in `attributes`. - **TraceContext** retains the field structure of `ExecutionTraceContext`. `inputRunId`, `businessKey`, and `agentName` retain their existing meanings. The changes extend the context to describe Events: - For Event records, `entityType` is `event` and `entityName` is the Event's type. - `executionId` and `parentExecutionId` retain their execution semantics and are absent from Event records. An Event record refers to its producer through `entityMetadata.producerExecutionId`. - `entityMetadata` holds Event identity and source information (`eventId`, `producerExecutionId`, `upstreamEventId`, and `upstreamActionName`) on Event records. Action records gain `triggerEventId` to identify the Event that triggered the execution. - **TraceRecord** contains its TraceContext. Serialization places context fields such as `entityType` and `inputRunId` at the top level of the JSON record. - **EventContext** remains part of the EventListener callback contract and is fully independent of TraceContext. There is no containment, inheritance, or conversion relationship between the two, and TraceRecord construction does not depend on EventContext. Payload truncation applies only to `attributes`. IDs, relationship fields, and the top-level `problemCategory` remain intact, preserving the information needed to connect records and classify failures. Truncation affects only the serialized content; the Event delivered to user code is unchanged. #### Representing an Event An Event observation is a standalone TraceRecord. Its context identifies the Event, its attributes contain the Event payload, and its metadata links it to the producing Action execution when one exists. Consider a `create_order` Action execution (`action-1`) that consumes Event `event-1` and emits an `OrderCreated` Event (`event-2`). The examples below use abbreviated IDs and omit timestamps and common metadata for brevity. **Current Event Log record, with tracing enabled.** The record combines the output Event with its producer's execution context: `entityType`, `entityName`, and `executionId` describe `create_order`, while the `event*` fields describe `OrderCreated`. ```json { "inputRunId": "run-1", "entityType": "action", "entityName": "create_order", "executionId": "action-1", "eventId": "event-2", "eventType": "OrderCreated", "upstreamEventId": "event-1", "upstreamActionName": "create_order", "eventAttributes": { "orderId": "order-1" } } ``` **Proposed TraceRecord.** The context now describes `OrderCreated` itself. Its relationship to the `create_order` execution is explicit in `producerExecutionId`. ```json { "inputRunId": "run-1", "entityType": "event", "entityName": "OrderCreated", "entityMetadata": { "eventId": "event-2", "producerExecutionId": "action-1", "upstreamEventId": "event-1", "upstreamActionName": "create_order" }, "attributes": { "orderId": "order-1" } } ``` The Event's `type` maps to `entityName`, its `id` to `entityMetadata.eventId`, and its payload directly to `attributes`. This preserves the Event's identity and content without embedding the Event object in the record. Event flow and execution relationships use the following fields: | Relationship | Representation | |---|---| | An Action invokes an LLM, Parser, or Tool | The child execution's `parentExecutionId` points to the Action execution | | An execution emits or replays an Event | The Event record's `entityMetadata.producerExecutionId` identifies that execution | | An Event triggers an Action execution | The Action record's `entityMetadata.triggerEventId` identifies that Event | | `create_order` consumes `event-1` and emits `event-2` | The `event-2` record's `entityMetadata.upstreamEventId` is `event-1`, and its `upstreamActionName` is `create_order` | These relationships preserve a distinction between an Event's identity and the execution that produces or replays it: - An Event has no execution lifecycle. Its record therefore omits `executionId`, `parentExecutionId`, and top-level `status` and `problemCategory`. User attributes with these names remain in `attributes` and are subject to payload truncation. - `producerExecutionId` is absent when there is no producing Action execution, as with a root InputEvent. Framework-generated Events retain their existing source information even when no Action execution can be referenced. - `eventId` identifies the Event, not a unique observation. The same Event may appear in multiple records, including when a saved output is replayed during recovery. - On replay, `producerExecutionId` identifies the execution emitting the saved output at that point. This may differ from the execution that originally produced it. - `executionId` retains its existing task creation and restoration semantics. A restart does not necessarily assign a new execution ID. Source fields move from the programming-model Event to Trace metadata. Custom Event construction and reconstruction continue to use the Event's ID, type, and attributes. ### 3.2 Collection and Runtime Flow Integrating TraceRecord into the runtime requires three changes: constructing records directly at the existing collection points, moving Event source information into those records, and retaining the context needed to connect records independently of logging configuration. #### Current flow Records currently reach EventLogWriter through three paths: - **Events:** EventRouter supplies the Event, EventContext, and optional ExecutionTraceContext before Action matching or downstream delivery. This path covers input, output, custom, and framework-generated Events, including Events with no consumers. - **Action lifecycle:** ActionExecutionOperator reports start, completion, failure, and result reuse by creating lifecycle Events and passing them through ExecutionEventLogger. - **Component calls:** Existing LLM, Parser, and Tool call sites report through ExecutionReporter and RunnerContext, which supply execution context and create lifecycle Events. Python reports use the existing Python-to-Java bridge. #### Runtime changes ##### 1. Construct TraceRecords at the collection points Each collection point will produce a TraceRecord describing the Event or execution it observes. Execution reports no longer need a synthetic Event to carry their status and content. | Observation | Current construction | Proposed construction | |---|---|---| | Event | EventLogRecord combines the Event, EventContext, and optional execution context. | EventRouter constructs a TraceRecord with the Event's identity, content, and available run and source references. | | Action or component execution | Reporting code creates a lifecycle Event and combines it with execution context. | Reporting code constructs a TraceRecord with execution context, status, and any failure details. | All records then enter a common filtering, serialization, and output path governed by Trace Log configuration. Existing reporting methods can retain their signatures while their implementations construct TraceRecords directly. ##### 2. Populate Event relationships from runtime context Today, the runtime writes `upstreamEventId` and `upstreamActionName` onto an emitted Event and supplies its producer's execution context to the logger. Under the new model, the runtime places these relationships in Trace metadata: - When creating an Action task, it records the triggering Event's ID in the Action's TraceContext as `entityMetadata.triggerEventId`. - When constructing a TraceRecord for an Event emitted by that Action, it reads the triggering Event ID, Action name, and execution ID from the current task. These become `upstreamEventId`, `upstreamActionName`, and `producerExecutionId` in the Event record's `entityMetadata`. - Component calls continue to use child execution contexts, retaining their execution IDs and parent execution references. ##### 3. Preserve context independently of record filtering Omitting an execution record must not remove the context needed to describe its output Events. For example, an `OrderCreated` record still references the `create_order` execution through `producerExecutionId` even when that Action's execution records are not written. The runtime therefore retains the required TraceContext through asynchronous resumption and result replay, regardless of which records the logger selects. A replayed Event record references the execution replaying it, following the identity semantics described in Section 3.1. #### Preserved behavior - **Collection timing and coverage:** Events are observed before Action matching or downstream delivery. Memory observations are collected during an Action and emitted as Events when it completes; the optional run-begin Event captures initial short-term Memory before the input's Actions execute. Framework Event generation conditions and payloads are unchanged. - **Routing and callbacks:** Event routing and EventListener delivery retain their existing behavior. Listeners continue to receive EventContext and Event through a separate callback path. Execution reports do not enter Action routing or trigger EventListener callbacks. - **Execution and recovery:** Action execution and component invocation retain their existing behavior, as do recovery and result reuse within the same version. Saved output Events continue to re-enter EventRouter after the Action reuse report. Trace reporting and output remain best effort and do not change Action execution or recovery guarantees. ### 3.3 Configuration All logging options move under `trace-log.*`, covering three areas: 1. **Record selection:** choose which Events and executions to log. 2. **Content detail:** truncate large payloads or retain them in full. 3. **Output destination:** write records through SLF4J or to files. Global settings establish the defaults. `trace-log.entity-levels` sets logging levels for particular Event types, Actions, or component calls, overriding the defaults for matching records. #### Configuration mapping | Existing option | Proposed option | Behavior and default | |---|---|---| | `event-log.trace.enabled` | `trace-log.default-scope` | Selects Event records (`EVENT_ONLY`) or all supported record types (`ALL`) by default. Local settings can override this selection. Default: `EVENT_ONLY`. | | `event-log.level` | `trace-log.level` | Sets the default level for selected records: `OFF` omits them, `STANDARD` applies payload limits, and `VERBOSE` retains full payloads. Default: `STANDARD`. | | `event-log.type.<EVENT_TYPE>.level` | `trace-log.entity-levels` | Sets logging levels by `entityType` and optional `entityName`. Ordinary Event-type overrides use `entityType: event` and the Event type as `entityName`. | | `event-log.standard.max-string-length` | `trace-log.standard.max-string-length` | Maximum retained string length at `STANDARD`. Default: `2000`. | | `event-log.standard.max-array-elements` | `trace-log.standard.max-array-elements` | Maximum retained array elements at `STANDARD`. Default: `20`. | | `event-log.standard.max-depth` | `trace-log.standard.max-depth` | Maximum retained nesting depth at `STANDARD`. Default: `5`. | | `eventLoggerType` | `trace-log.output.type` | Selects `SLF4J` or `FILE`. Default: `SLF4J`. | | `baseLogDir` | `trace-log.output.base-dir` | A non-empty value selects file output and takes precedence over `trace-log.output.type`, preserving existing behavior. | | `prettyPrint` | `trace-log.output.pretty-print` | Enables multiline JSON formatting. Default: `false`. | - Payload limits apply only to `attributes` at `STANDARD`. `VERBOSE` retains the full payload. A limit of `0` removes that particular limit without disabling logging. - Output remains JSONL by default, with one JSON record per line. Pretty printing retains the existing multiline JSON format. - Memory Event and run-begin Event options continue to control Event generation independently of logging. Trace settings determine whether the resulting Events are recorded. The `event-listeners` setting is unchanged. #### New capabilities and their purpose 1. **A default scope for common use cases.** `trace-log.default-scope` provides a starting point that settings for specific Events or executions can refine. With `trace-log.level: STANDARD`: - `EVENT_ONLY` records Events by default. Action and component execution records require a matching entry in `trace-log.entity-levels`. - `ALL` records Events and executions by default. Local `OFF` settings can exclude specific records. Users can begin with `EVENT_ONLY`, add records for one Action, and suppress a noisy Event type without an additional Trace switch. The default combination of `EVENT_ONLY` and `STANDARD` preserves today's default Event logging and payload limits. Event records also carry available run and producer references, regardless of whether execution records are enabled. Event-only logs capture the flow through Actions that emit Events. They cannot show an Action that emits no Event; setting a level for that Action can include its execution records for diagnosis. 2. **Logging levels for specific Events and executions.** Existing per-type level settings match an Event's `type`, which exposes synthetic lifecycle Event names when filtering execution reports. `trace-log.entity-levels` instead assigns levels using the descriptive fields on TraceRecord. Each entry uses the following fields: - `entityType` is required: for example, `event` selects Event records and `action` selects Action execution records. - `entityName` is optional and supports exact or prefix matching. It identifies the Event type or execution name, such as an Action's name. Omitting it matches all records of the specified `entityType`. - `level` is optional and controls whether matching records are written and how much payload is retained. If omitted, it inherits the global `trace-log.level`. Names match exactly unless prefix matching is explicitly requested. Event-type prefixes preserve the existing dot-separated hierarchy: a prefix of `com.foo` matches `com.foo` and `com.foo.OrderCreated`, but not `com.foobar.OrderCreated`. The configuration syntax for explicit prefixes will be specified separately. The same entry structure applies to LLM, Parser, and Tool executions. Matching by Agent name, business key, status, or problem category is deferred. For example, the following configuration retains Event logging, suppresses `DebugEvent` records, and enables verbose execution logging for `create_order`: ```yaml trace-log.default-scope: "EVENT_ONLY" trace-log.level: "STANDARD" trace-log.entity-levels: - entityType: event entityName: DebugEvent level: "OFF" - entityType: action entityName: create_order level: "VERBOSE" ``` | Record | Effective behavior | |---|---| | `DebugEvent` | Omitted by its local `OFF` setting. | | Other Events | Written at `STANDARD`, following the global defaults. | | The `create_order` Action | Execution records written at `VERBOSE`, following its local setting. | | Other Actions and component calls, including calls inside `create_order` | Execution records omitted unless another entry selects them. Execution itself is unaffected. | Each entry applies to matching records. Setting a level for `create_order` does not also set the level for its output Events or child calls. In this example, `OrderCreated` follows the Event defaults, while LLM and Tool calls inside `create_order` require separate matching entries. #### Level selection and precedence For each record, the logger selects the most specific matching entry in `trace-log.entity-levels`. Matching entries take precedence over global defaults in the following order: 1. Exact `entityType` and `entityName` match. 2. Matching name-prefix entry within the entity type, with the longest prefix winning. 3. An `entityType`-only entry. 4. The global `trace-log.default-scope` and `trace-log.level`, if no entry matches. The selected entry determines the record's level: `OFF` omits it, `STANDARD` applies payload limits, and `VERBOSE` retains the full payload. An entry without an explicit level inherits `trace-log.level` directly; it does not inherit from a less specific entry. When no entry matches, the global settings apply: - Under `EVENT_ONLY`, Event records use `trace-log.level`; execution records are omitted. - Under `ALL`, both Event and execution records use `trace-log.level`. Local settings can therefore both exclude records selected by the defaults and include records outside the default scope. A global level of `OFF` disables logging by default, while explicit local `STANDARD` or `VERBOSE` settings can still enable it for selected records. Entry order in the configuration has no effect on precedence. Conflicting entries of equal specificity are rejected at startup. The runtime reads and validates configuration at startup and reports the effective defaults and entries. These semantics are consistent across Java, Python, and YAML. #### Configuration migration - **Legacy keys require explicit migration.** Any recognized old logging key causes startup to fail with migration guidance, including when old and new keys are mixed. - **Global settings follow the mapping above.** Set `trace-log.default-scope` to `EVENT_ONLY` when migrating from `event-log.trace.enabled: false`, or to `ALL` when migrating from `true`. Migrate the level, payload limits, and output settings at the same time. - **Ordinary Event-type overrides become entries in `trace-log.entity-levels`.** Use `entityType: event` and preserve the exact-name or prefix matching behavior of the original setting. - **Synthetic lifecycle filters have no general equivalent.** An existing per-type setting may suppress only `_execution_finished_event` while retaining start and failure records. A level configured for an Action or Tool applies to all lifecycle statuses for that execution, so it cannot reproduce this behavior. Status-based matching is deferred, and migration guidance must state this limitation explicitly. Rejecting legacy keys prevents an old configuration that disabled logging from silently falling back to the new defaults and emitting records. ### 3.4 Output and Consumption Both output destinations serialize the same TraceRecord format. The output layer retains the resolved `logLevel` and existing job, task, and subtask information: SLF4J includes `jobId`, `taskName`, and `subtaskId` in each record, while file output identifies them in the file path. This metadata is added by the output layer, independently of the Event and execution context described in Section 3.1. #### Field mappings for readers and queries Queries and parsers use the following mappings to read the new format: | Information | Current representation | Proposed representation | |---|---|---| | Event identity and content | `eventType`, `eventId`, and `eventAttributes`. | For `entityType = "event"`, the type is `entityName`, the ID is `entityMetadata.eventId`, and the content is `attributes`. | | An output Event's source | Top-level `upstreamEventId` and `upstreamActionName`. | The same fields in `entityMetadata`. | | Execution progress | Execution fields, synthetic lifecycle Event types, and status. | `entityType`, `entityName`, and `executionId` describe the execution; `status` describes its lifecycle state. | Custom queries and parsers must adopt these mappings. Support for historical formats in the built-in reader does not extend automatically to external tools. #### Built-in Trace Tree support The Trace Tree tool will read JSON record files produced by Trace Log and continue to build the existing Event–Action graph from their Event records. Execution records remain outside its graph construction; reading the new format does not add execution-state or component-call visualization. - The reader accepts the new TraceRecord format, current flat Event Log records, and older records with a nested `event` object. - New records are classified by `entityType`; an Event's name does not make it an execution record. Historical records retain the existing lifecycle-report recognition behavior. - The graph's output structure and existing reconstruction behavior remain unchanged. Event IDs and source references continue to provide the relationships used to build it. - Historical records are interpreted using the information they contain. The reader does not invent missing execution IDs or relationships. ## 4. Compatibility and Migration Boundaries The migration affects log producers, readers, configuration, and some runtime structures. The following boundaries distinguish changes to observability from the contracts retained by the programming model. | Interface or stored data | Compatibility boundary | |---|---| | EventListener | Callback signatures, timing, and EventContext behavior are unchanged. Trace settings do not affect callback delivery or the Event payload received by listeners. | | Custom Events | Construction and reconstruction retain the Event's ID, type, and attributes. Code that directly accesses the removed upstream fields must be updated, including EventListener implementations that read those fields. | | Event JSON | Event deserialization in Java and Python continues to accept older Event JSON containing `upstreamEventId` and `upstreamActionName`. These fields are ignored; the Event's ID, type, and attributes are preserved. This compatibility does not extend to runtime state restoration across versions. | | Internal reporting helpers | ExecutionTraceContext, lifecycle Event factories, and reporting internals can be refactored without a separate deprecation period for each internal helper. | | Log format | New versions write TraceRecord. Built-in readers also support historical formats; external queries and parsers require migration. | | Operational naming | Loggers, log files, and related metrics adopt Trace naming. Existing log collection and monitoring configurations must be updated accordingly. | | Configuration | Legacy logging keys are rejected at startup with migration guidance, as described in Section 3.3. | | Persisted runtime state | Checkpoints and savepoints may contain the previous ActionTask and Event structures; ActionStateStore also persists triggering and output Events for result reuse. This proposal does not introduce cross-version migration for those structures. Upgrade guidance must identify affected restore and result-reuse paths; recovery and result reuse within the same version retain their existing behavior. | Trace remains a best-effort account of runtime activity. A missing record does not establish that an Event or execution never occurred, and repeated records with the same Event ID may describe repeated observations of that Event. These limits apply to the logs; business execution and recovery retain their existing guarantees. GitHub link: https://github.com/apache/flink-agents/discussions/1146 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
