goutamadwant opened a new issue, #11364: URL: https://github.com/apache/seatunnel/issues/11364
### Search before asking - [x] I had searched in the [feature](https://github.com/apache/seatunnel/issues?q=is%3Aissue+label%3A%22Feature%22) and found no similar feature requirement. ### Description This is a STIP/design proposal for #11354. SeaTunnel CDC supports different source families and offset models, including MySQL binlog positions and GTIDs, PostgreSQL LSNs, Oracle SCNs, TiDB resolved timestamps, and connector-specific split or restore state. Today these signals are not exposed through one consistent CDC progress model. Operators often need to inspect connector-specific logs, internal offsets, or checkpoint details to understand whether a CDC job is still snapshotting, catching up, stalled, or correctly resumed after restore. The goal of this STIP is to define a unified CDC lag and progress observability model without hiding connector-native details or pretending that all CDC connectors support the same exact lag semantics. The proposed direction is: - define a normalized CDC progress snapshot model - separate snapshot progress from incremental lag - preserve connector-native raw offset details - make restore and resume progress visible - document required, optional, and unsupported fields per connector family - expose the model through an API or existing job detail path first - add UI support after the model/API shape is agreed Proposed top-level model fields: | Field | Description | | --- | --- | | jobId | SeaTunnel job id | | pipelineId | Pipeline id when available | | vertexId | Source vertex id | | connectorType | CDC connector type, for example MySQL CDC or PostgreSQL CDC | | tablePath | Table path when the connector can report table-level progress | | splitId | Split id when the connector can report split-level progress | | phase | Current CDC phase | | snapshotProgress | Snapshot progress details | | incrementalProgress | Incremental log/cursor progress details | | checkpointProgress | Checkpoint and restore progress details | | rawPosition | Connector-native offset fields | | supportLevel | Exactness level for reported fields | | lastProgressTime | Last time progress was observed | Suggested phase values: | Phase | Meaning | | --- | --- | | SNAPSHOT | The connector is reading initial table or split data | | CATCH_UP | Snapshot is complete, but the connector is still catching up to a known source watermark | | INCREMENTAL | The connector is consuming live change events | | RESTORING | The connector restored from checkpoint state and has not yet reported steady progress | | STALLED | The connector or engine can confidently detect no progress for a configured or observed interval | | UNKNOWN | The connector cannot safely report the current phase | Snapshot progress should be represented separately from incremental lag. Suggested snapshot fields: | Field | Description | | --- | --- | | assignedSplitCount | Number of snapshot splits assigned | | completedSplitCount | Number of snapshot splits completed | | runningSplitCount | Number of snapshot splits currently running | | remainingSplitCount | Number of snapshot splits not completed yet | | currentTable | Current table if table-level progress is available | | currentSplitId | Current split if split-level progress is available | | lowWatermark | Connector-native low watermark if available | | highWatermark | Connector-native high watermark if available | | supportLevel | Exactness level for snapshot progress | Incremental progress should describe log or cursor consumption. Suggested incremental fields: | Field | Description | | --- | --- | | currentConsumedPosition | Current consumed CDC position | | sourceHighWatermark | Source-side high watermark if available | | normalizedLag | Lag value if the connector can report it safely | | lagUnit | Unit of the normalized lag | | lastSourceEventTime | Last source event timestamp if available | | lastConsumedEventTime | Last consumed event timestamp if available | | lastProgressTime | Last time incremental progress was observed | | supportLevel | Exactness level for incremental progress | Suggested lag units: | Unit | Meaning | | --- | --- | | ROWS | Lag is represented as row count | | EVENTS | Lag is represented as event count | | BYTES | Lag is represented as bytes | | TIME_MS | Lag is represented as time delay | | OFFSET_DISTANCE | Lag is represented as connector-native offset distance | | UNKNOWN | Lag exists but cannot be normalized safely | Checkpoint and restore progress should help verify whether a CDC job resumed from the expected checkpointed position. Suggested checkpoint and restore fields: | Field | Description | | --- | --- | | lastCheckpointId | Last completed checkpoint id | | lastCheckpointedPosition | CDC position stored in the last completed checkpoint | | restoredFromPosition | CDC position restored after restart, failover, or upgrade | | currentConsumedPositionAfterRestore | First or latest consumed position after restore | | restoreTime | Time when the restore was observed | | supportLevel | Exactness level for restore progress | Raw connector position examples: | Connector | Raw fields | | --- | --- | | MySQL CDC | binlogFile, binlogPosition, gtidSet, serverId | | PostgreSQL CDC | lsn, txId, commitTimestamp | | Oracle CDC | scn, commitScn | | TiDB CDC | resolvedTs, checkpointTs | Support levels: | Support level | Meaning | | --- | --- | | EXACT | The connector can report this field with precise semantics | | BEST_EFFORT | The connector can report an approximate or partially complete value | | UNSUPPORTED | The connector cannot report this field safely | Initial connector scope: - MySQL CDC - PostgreSQL CDC - Oracle CDC - TiDB CDC Non-goals for the first version: - do not force every connector to expose identical raw offset fields - do not define one universal numeric lag if the connector cannot support it correctly - do not replace connector logs for deep debugging in the first version - do not add a long-term historical metrics store in the first version - do not change CDC restore behavior as part of this STIP Open questions: - Should the model live in `connector-cdc-base`, `seatunnel-api`, or an engine/API module? - Should this be exposed through existing metrics, a new CDC progress endpoint, or both? - How much recent progress history should be kept, if any? - Should `STALLED` be computed by connectors or by the engine/API layer? - Which fields should be required for phase 1 connectors? - Should support levels apply per field or per field group? - How should this interact with future upgrade-compatibility checks? ### Usage Scenario This is mainly for operators running long-lived CDC jobs who need to understand CDC progress and restore correctness without reading connector-specific logs or internal checkpoint state. Example scenarios: - A MySQL CDC job is running for many tables and the user wants to know whether the job is still snapshotting, catching up, or consuming live binlog events. - A PostgreSQL CDC job appears delayed and the user wants to compare the consumed LSN with the latest source-side progress. - An Oracle CDC job resumes after failure and the user wants to verify that it restored from the checkpointed SCN instead of an older startup position. - A TiDB CDC job has resolved-ts or checkpoint-ts movement issues and the user needs a product-level view of whether progress is advancing. - A multi-table CDC job has uneven progress and the user wants to see which table or split is lagging. - After upgrade or failover, the user wants to verify that SeaTunnel resumed from the expected CDC position. The expected user-facing result is not one universal lag number. The expected result is a consistent CDC progress view that shows: - current phase - snapshot progress - incremental cursor/log progress - checkpointed and restored position - raw connector-native offset details - exact, best-effort, or unsupported support level per connector field This lets users compare CDC progress across connectors while still keeping the native offset details needed for diagnosis. ### Related issues _No response_ ### Are you willing to submit a PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
