goutamadwant opened a new issue, #13107: URL: https://github.com/apache/seatunnel/issues/13107
### Problem #12983 needs a CDC job to change schema, checkpoint, restart and emit correctly typed restored rows. #12776 adds Spark CDC startup support, but does not provide the durable source-state and typed-query recovery protocol needed for that test. Rebuilding InternalRowCollector alone cannot restore the source position or reanalyze the query, transforms and sink. ### Reproduction and proposed behavior Use a multi-table MySQL CDC job, add a column, complete a checkpoint, stop the job, apply another DDL change and restart. Check the first restored values and sink columns/types. Repeat with schema propagation disabled. **Before:** the #11932/#12776 integration base lacks this recovery protocol; focused restore probes also exposed converter mismatches. **With the prototype stack:** source intervals, committed offsets and schema phases survive restart. Work using the previous schema commits before the typed query, transforms and sink are rebuilt. Completed physical DDL is reconciled before replay. ### Validation The runtime branch passed 149 focused tests and full Java 11 repository verify with tests skipped. The collector composition passed 154 focused tests in both default and Java 8 profile builds. [Spark restore fixture](https://github.com/goutamadwant/seatunnel/blob/03225b72218a24c7428b556bed2162711ee4359c/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/java/org/apache/seatunnel/connectors/seatunnel/cdc/mysql/MysqlCDCWithSparkSchemaRecoveryIT.java) passed all four native MySQL cases across Spark 2.4.6/3.3.0: enabled/disabled propagation, live/offline DDL, exact restored columns/types/values and insert-only audit after runtime shutdown. Tests used default 180-second assertion timeouts and unmodified test methods. Spark 3 used an architecture-matched Java 11 image. ### Scope and discussion Two prerequisite PRs belong here: durable recovery and Spark 2 Java 8-compatible packaging. The collector refresh/acceptance patch stays in #12983. The proposed contract is source replay with at-least-once sink behavior, one sink query and stable checkpoint ownership/routing/parallelism. XA/stateful sinks requiring writer-state recovery are rejected before startup. Native PostgreSQL, secured/HA HDFS, default ARM images and hosted CI are not established. Please agree on the recovery contract, checkpoint compatibility, API boundaries and whether a STIP is appropriate before upstream implementation review. Related: #12983, #11932 and #12776. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
