goutamadwant commented on issue #11354: URL: https://github.com/apache/seatunnel/issues/11354#issuecomment-4887006031
@davidzollo I would like to take this up, starting with the STIP/design first before any implementation. The design approach I want to propose is to treat CDC lag as a CDC-owned progress model, not just another generic job metric. The main reason is that MySQL binlog position / GTID, PostgreSQL LSN, Oracle SCN, and TiDB resolved-ts do not mean the same thing. If we expose only one generic lag number, it can easily become misleading. My rough design direction is: **1. Add a normalized CDC progress snapshot model** The model should describe the job's CDC progress without hiding connector-native details. Fields I am thinking about: - jobId / pipelineId / vertexId - connector type - table or split id where meaningful - phase: snapshot, catch-up, incremental, restoring, stalled, unknown - current consumed position - source high watermark position where available - last checkpointed position - restored-from position after restart or upgrade where available - last progress timestamp - normalized lag fields where meaningful - raw connector position fields as a map - support level per field: exact, best-effort, unsupported The important part is to separate normalized progress from raw connector-specific position. **2. Connector adapters should produce the CDC progress** Each CDC connector should translate its native state into this model: - MySQL CDC: binlog file/position, GTID if available - PostgreSQL CDC: LSN - Oracle CDC: SCN - TiDB CDC: resolved timestamp / checkpoint timestamp The engine should not need to understand how to compare every database-specific offset. The connector should provide the best available progress snapshot and mark unsupported fields clearly. **3. Snapshot progress and incremental lag should stay separate** Snapshot phase should report table/split progress and completed snapshot splits. Incremental phase should report log/cursor progress and lag from the source high watermark if the connector can provide it. This avoids mixing "snapshot still running" with "incremental stream is behind", since those are different operational states. **4. Restore/resume should be visible** For restore and upgrade debugging, I think the model should include: - restored-from position - latest checkpointed position - current consumed position after resume This would help verify that a CDC job resumed from the expected checkpoint instead of falling back to an older startup position. **5. API first, UI second** For the first implementation phase, I would prefer to define the model and expose it through an API or existing job detail path first. UI can then consume the same contract, but I would not start with UI-first work because the hard part is agreeing on the CDC progress contract. **6. First connector coverage** For phase 1, I would keep the connector set representative but not exhaustive: - MySQL CDC - PostgreSQL CDC - Oracle CDC - TiDB CDC That covers the main offset families and should be enough to validate the model. **7. Non-goals for the first version** - no single universal lag number without semantics - no forcing every connector to expose identical raw fields - no long-term history store - no restore behavior change - no large UI redesign before the model/API is agreed If this direction sounds reasonable, I will draft the STIP around this phased design and wait for maintainer feedback before opening implementation PRs. Let me know your thoughts and any guidance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
