DanielLeens commented on issue #11381: URL: https://github.com/apache/seatunnel/issues/11381#issuecomment-4920569115
This is a valuable direction, but it should be handled as an engine-level design topic rather than as a connector-only follow-up. The core concern is real: a connector-side flush like PR #11156 helps Doris, but by itself it does not create a single ordering contract for every `SchemaChangeEvent` path. If a schema event can still reach sinks before old-schema buffered data has crossed a durable checkpoint / flush boundary, the engine is still relying on per-connector correctness in a place that should be much more explicit. Before implementation starts, I think the next revision should make four boundaries clearer: 1. which schema-change events must always enter this drain protocol in v1, and whether `CREATE TABLE` is intentionally excluded 2. what exact engine condition means "old-schema data is drained" for sink subtasks, especially across concurrent checkpoints, retries, and failover 3. how this interacts with connectors that already have their own schema-change checkpoint behavior or transactional commit model 4. what compatibility contract this creates for Source -> Transform -> Sink paths that emit schema changes outside the current CDC collector flow The implementation direction itself looks reasonable: reuse the existing schema-change checkpoint lifecycle instead of calling sink commits out of band, and gate sink-side `applySchemaChange(...)` until the pre-change drain phase is complete. I would strongly suggest keeping the first version narrow around one enforceable contract: old-schema records must cross a checkpoint / flush boundary before downstream DDL is applied. If those boundaries are tightened, this looks like a good design-track issue to continue, and it may even be worth expressing the implementation plan in a small STIP-style follow-up before code changes begin. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
