nzw921rx commented on issue #11364: URL: https://github.com/apache/seatunnel/issues/11364#issuecomment-4965864823
@goutamadwant Thanks for the detailed update. The new sections clarify several of the lifecycle concerns raised earlier, and the overall direction is becoming much easier to follow. I think a few runtime-contract details would still benefit from further clarification before the implementation moves forward. The remaining questions are less about individual fields and more about how reports are produced, transported, aggregated, and interpreted. 1. **Reporting path** `CdcProgressProvider#getCdcProgress()` is currently described as a local getter. Since the reader runs on a worker and the enumerator runs on the master, could we please clarify: * which engine component invokes the provider; * whether it is invoked at specific lifecycle points or on a sampling interval; * how invocation is coordinated with `pollNext`, `run`, and checkpoint callbacks; * how reports are transported across the worker/master boundary; * how updates are coalesced before being written to the latest-snapshot store. Clarifying this path would help show how the connector-facing capability becomes an engine-visible API response. 2. **Report scope and aggregation** The enumerator owns snapshot assignment state, while readers own consumed positions. It may help to define the responsibilities and aggregation rules for: * enumerator-level reports; * reader-task reports; * split-level details; * source/vertex-level summaries. For parallel readers, a singular top-level `phase`, `tablePath`, or `splitId` may not fully represent the runtime state. It would also be useful to explain how missing, stale, or temporarily inconsistent task reports are handled. 3. **Restore origin** `SourceReader.addSplits(...)` is used for both normal split assignment and restored state. Because of this, it may be difficult for a reader to determine from `addSplits` alone whether a split came from restore. Could the design introduce an explicit engine-provided restore context or lifecycle callback containing the restored checkpoint identity and restored split positions? This would make the restore semantics less dependent on inference inside the connector. 4. **Lifecycle and health status** It may be clearer to keep these concepts separate: * `SNAPSHOT`, `CATCH_UP`, and `INCREMENTAL` as CDC lifecycle stages; * `RESTORING` as a recovery transition; * `STALLED` and backpressure as health or runtime conditions. It would also help to document the exact MySQL condition for the `CATCH_UP -> INCREMENTAL` transition, rather than only listing the available phase values. 5. **Time and stalled semantics** Could the model distinguish at least: * report observation time; * last position-change time; * last source-event time; * source-high-watermark observation time. An unchanged offset can mean several different things: an idle source, downstream backpressure, checkpoint coordination, or an actual lack of reader progress. Without evidence that the source head is ahead of the consumed position, it may be safer for the framework to report “no progress observed” or “stale report” rather than concluding that the CDC source is stalled. 6. **Final API shape** The issue currently keeps some earlier field definitions and lets the later clarification section take precedence. To make the contract easier to review and implement, could these sections eventually be consolidated into one canonical DTO or response schema covering: * connector-owned reports versus the engine-enriched view; * position type and schema version; * nullability and phase/payload invariants; * support and accuracy representation; * task and split cardinality; * REST payload limits or pagination. Group-level support may also need another look. With lazy chunking, for example, assigned and completed counts may be exact while the final split total and overall percentage remain unknown. One possible implementation sequence would be to validate an internal model through a complete MySQL reader/enumerator/engine/REST path first, then use PostgreSQL to validate a second offset family, and finally stabilize the public `seatunnel-api` contract based on those two runtime paths. A compact end-to-end sequence diagram together with the final response schema would probably resolve most of these remaining questions and make the implementation slices easier to review. Thanks again for continuing to refine the proposal. The overall feature is valuable, and clarifying these runtime boundaries should help reduce rework as the implementation progresses. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
