DanielLeens commented on issue #13107: URL: https://github.com/apache/seatunnel/issues/13107#issuecomment-6595076666
Thanks for opening this as a discussion before sending code. Some `dev` facts to anchor it: - On `dev`, CDC sources do not run on Spark at all. `IncrementalSource` does not implement `SupportCoordinate`, so it takes the parallel path, and `ParallelReaderContext#sendSourceEventToEnumerator` throws `UnsupportedOperationException`. - #12776 adds startup support only. It explicitly documents "no checkpoint restore", rejects micro-batch reads of coordinated sources, and currently has merge conflicts. - The Spark 3.3 micro-batch keeps its offset in memory (`SeaTunnelMicroBatch`, `currentOffset` starts at `SeaTunnelOffset.of(0L)` and is advanced in `commit`). Reader state is reloaded from `checkpointId - 1` on the configured HDFS path (`ParallelMicroBatchPartitionReader` L88). So durable restart recovery on Spark is a new capability, not a fix to existing behaviour. I agree a STIP is the right vehicle. It would define a Spark-specific delivery contract and a new persisted state format, and it would reject some sinks. Implementation PRs should wait until that is agreed and #12776 has landed. Points the STIP needs to settle: 1. Delivery contract: "source replay + at-least-once sink" is weaker than exactly-once on Zeta. How is it surfaced to users (docs plus an explicit startup error)? Is it opt-in or the default for CDC on Spark? 2. Sink admission: the exact rule for rejecting XA or writer-state sinks before startup, so that `is_exactly_once = true` JDBC is never silently downgraded. 3. State ownership and format: where source intervals, committed offsets and schema phases live (Spark `checkpointLocation` vs the SeaTunnel HDFS path), how they are versioned, and what happens when parallelism, the table set or the job config changes between runs (reject vs migrate). 4. Schema phase: a running Structured Streaming query cannot change its output schema. Define how rows written under the previous schema are committed before the query is rebuilt, and how DDL applied while the job was stopped is reconciled without replaying duplicate downstream DDL. This matches acceptance item 3 in #12983. 5. Spark 2.4 / Java 8 packaging: please justify it in its own PR, with the dependency and size impact. 6. Validation: the four native MySQL cases should run in hosted CI on the final heads, for both Spark 2.4.6 and 3.3.0. The collector refresh stays in #12983. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
