joeyutong opened a new pull request, #1219: URL: https://github.com/apache/flink-agents/pull/1219
Fixes #1154 Design: https://github.com/apache/flink-agents/discussions/1146 ### Purpose of change Events and Action/component calls now use one structured Trace Log. Event payloads are separated from identity and source relationships, and call outcomes are recorded directly instead of being represented as synthetic Events. This makes tracing independent of Event routing and gives users one configuration model for selection and detail. #### Runtime flow The runtime creates Event observations and Action/call reports with their run and relationship context. Metrics consume reports independently of logging. `TraceLogWriter` resolves each record's target and detail once, skips unselected records, and passes selected records to the FILE or SLF4J logger. The logger sanitizes media, applies attribute limits at STANDARD, and writes JSON. The CLI reads the new format and reconstructs the existing Event tree. #### Key decisions - Each `trace-log.targets` entry combines `scope` and optional `detail`; presets and entity selectors use the same shape. - `detail` is OFF, STANDARD, or VERBOSE. Content limits affect attributes only, so recorded identities and relationships remain available. - Source relationships belong to trace metadata rather than mutable Event fields. Execution reports remain an optional capability on RunnerContext. ### Behavioral Semantics #### Interaction decisions | Conditions | Result | |---|---| | Targets omitted / explicitly empty | EVENT_ONLY at STANDARD / no records | | Several matching scopes | Exact name, longest `.*` namespace, type, EVENT_ONLY, then ALL; order-independent | | Broad OFF and more specific STANDARD/VERBOSE | The specific match enables recording | | Local detail omitted | Inherit the applicable preset, including OFF; do not inherit another entity selector | | Only a child entity selected | Record it with available relationship IDs; do not enable its parents | | STANDARD / VERBOSE | Bounded / full attributes; both sanitize media and preserve structural fields | #### Behavioral contracts Event identity remains stable when an Event is reconstructed or observed repeatedly. Action reports identify their triggering Event; component reports identify their containing Action and keep their call identity across lifecycle phases and continuation/recovery. Trace selection does not change Event listener callbacks, routing, or metric collection. Both outputs emit the same TraceRecord JSON structure with output metadata such as `detail`. The CLI merges observations of the same Event ID/type without treating payload truncation as an identity conflict. Its JSON tree shape remains unchanged. #### Failure behavior Malformed targets, conflicting duplicate scopes, and legacy logging configuration fail at initialization. Logger initialization and close errors propagate. Append/flush failures are counted and logged best-effort without failing the Action. Component failures retain failure classification and error attributes. The CLI reports malformed records and missing/conflicting relationships while retaining valid observations. ### Tests | Contract | Coverage | |---|---| | Identity, source metadata, and payload are distinct | `EventTest`, Python `test_event.py`, `TraceContextTest`, `TraceRecordJsonSerdeTest` | | Preset defaults, precedence, inheritance, local OFF, invalid configuration | `TraceLogTargetResolverTest`, Java/Python configuration tests, config-option parity check | | Selection once; best-effort write failure; listeners remain independent | `TraceLogWriterTest`, `EventRouterTest` | | Structural fields survive truncation; output and media handling | `FileTraceLoggerTest`, `Slf4jTraceLoggerTest`, `JsonTruncatorTest` | | Lifecycle identities and metadata survive replay/continuation | `RunnerContextImplExecutionReporterTest`, `ActionTaskContextManagerTest`, `ActionExecutionOperatorTest`, `EventLineageReconstructionTest` | | Metrics remain independent of trace selection | `BuiltIn*MetricsTest`, `ActionExecutionOperatorParallelMetricsTest` | | Java/Python reporter bridge and async/sub-agent integration | Python execution-reporter/RunnerContext tests; Java async, durable execution, and internal sub-agent suites | | CLI Event causality, reused observations, and malformed input | `test_trace_tree.py`; packaged PyFlink Trace Log-to-CLI test | Local checks: JDK 17 suites discovered 681 tests (666 passed, 15 engine-specific skips); JDK 21 integration suites discovered 186 (166 passed, 20 engine-specific skips). Python focused suites: 267 passed. Java/Python config parity, Spotless, Ruff, and Apache RAT passed. Packaged PyFlink 2.2.1 E2E: all 4 passed (Event output, runtime-to-CLI causality, memory Events, and cross-language media sanitization), using clean distribution artifacts and a rebuilt wheel. Not verified: external LLM/vector-store services, the full Flink/JDK and cross-language E2E matrices, production deployment, compatibility with historical logs/serialized Events, and cross-version checkpoint recovery. <details> <summary>Implementation invariants and validation scope</summary> The resolver precomputes per-type defaults and only performs cached name matching for types with named overrides. Loggers do not repeat target resolution. Event relationships use `eventId`, `producerExecutionId`, and `upstreamEventId` in entity metadata; Action metadata carries `triggerEventId`. Filtering retains those references even if the related records are omitted. Reporting state is scoped to each Action and transferred with continuation context. Lifecycle observations go directly to trace and metric consumers. The targeted Maven build compiles current API, plan, runtime, examples, and distribution artifacts. Python checks use the active environment's purelib path. Local PyFlink verification uses a wheel containing newly built JARs, rather than previously installed artifacts. </details> ### API This intentionally breaks the old Event Log configuration, logger extension interfaces, log schema, and Event source-field APIs in Java/Python. Custom output implementations use `TraceLogger.append(TraceRecord, TraceLogDetail)`. Historical Event JSON and log formats have no compatibility guarantee. Event IDs, payloads, attachments, Event listener callbacks, and ExecutionReporter method signatures are retained; the CLI continues to expose its existing Event-tree output structure. ### Documentation - [x] `doc-included` ### Was this patch authored or co-authored using generative AI tooling? - [x] Yes - [ ] No Generated-by: Codex desktop 155.0.8059.40 (GPT-6) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
