davidzollo opened a new issue, #11355: URL: https://github.com/apache/seatunnel/issues/11355
## Background SeaTunnel has been improving CDC schema evolution correctness in several areas, including policy handling, multi-table routing, comment-related schema changes, and failure handling. That work is important, but users still lack an operator-facing way to answer a simple question: **What exactly happened to my schema change, and why did the pipeline make that decision?** Today, when a DDL-related problem occurs, users often have to reconstruct the story from logs, connector behavior, sink capability assumptions, and trial-and-error. ## Problem to solve SeaTunnel does not yet provide a first-class **schema evolution timeline and decision log**. For production CDC jobs, operators should be able to inspect: - when a schema change event arrived - how the event was resolved to a table identifier/path - which behavior policy applied - whether the sink claimed support for that change type - whether the schema apply succeeded, was ignored, or failed - whether the event was acknowledged consistently across the pipeline Without this, schema evolution remains hard to debug even when the underlying behavior is technically correct. ## Proposed scope Add a schema evolution timeline and decision log for CDC jobs. Suggested first-delivery scope: - record and present the main lifecycle of a schema change event from source ingestion to sink-side handling - show the event type, affected table, resolved target, policy decision, and final outcome - make failures and ignored events explicit instead of leaving them implicit in logs - support multi-table paths where routing mistakes or ambiguity can occur - connect the timeline to the job detail experience so users can inspect recent DDL activity without leaving the runtime context ## Why this needs explicit design first This feature crosses multiple contracts: - CDC source event normalization - table resolution / routing - schema-change behavior policy - sink capability declaration and apply behavior - user-facing API/UI representation of those decisions If SeaTunnel exposes this feature without a clear design, it risks showing incomplete or misleading state about schema evolution. ## STIP requirement before implementation Because this feature introduces a cross-module decision/audit model for CDC schema evolution, **the claimant should submit a STIP design first and get maintainer agreement before starting implementation**. The STIP should explicitly define: - the event lifecycle model shown to users - which fields are stable product contracts versus debug-only details - how policy decisions are represented - how sink support, apply success, ignore, retry, and failure states are modeled - retention boundaries and how much schema-event history is kept - how the design interacts with multi-table sinks and table-resolution edge cases ## Acceptance criteria - Users can inspect recent schema evolution activity as a timeline instead of inferring it from logs. - The timeline clearly shows event type, target table, policy decision, and final outcome. - Failures and ignored events are distinguishable. - The first version documents scope limits and retention behavior clearly. - English and Chinese docs are updated. ## Non-goals for the first version - full payload archival for every raw DDL statement forever - a guarantee that every connector exposes identical schema-event detail on day one - automatic schema repair suggestions in the first delivery ## Related work - Issue #11043 - Issue #10776 - Issue #11016 - PR #11015 - PR #11025 - PR #11162 - PR #11328 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
