DanielLeens commented on issue #13157:
URL: https://github.com/apache/seatunnel/issues/13157#issuecomment-6687124960

   Thanks for spelling out the desired data and failure semantics. This should 
be tracked as a schema-evolution design item, not started as an isolated 
SQL-transform implementation.
   
   There are already two active, non-final pieces of the same end-to-end 
contract:
   
   - #12284 translates schema changes through the SQL transform, but it is 
Zeta-oriented, remains open, and its current Build is failing.
   - #11912 coordinates schema changes across parallel Flink sink writers, but 
it remains open, has a current conflict, and its latest Build timed out.
   
   Neither is proof that a MySQL CDC -> Flink SQL -> JDBC pipeline has a 
complete, recoverable contract. The umbrella discussion #11402 explicitly 
requires agreement on per-table event/data ordering, checkpoint boundaries, 
sink application and replay before the capability is declared complete.
   
   Please start with a short STIP/design proposal linked to #11402 and a 
`[email protected]` discussion. It needs to make these decisions 
testable:
   
   1. The exact first supported scope: source, SQL projection forms, JDBC sink, 
Flink versions, and the add/drop/rename/type-change operations that are 
accepted or rejected.
   2. The ordering boundary: old-schema records must drain before the schema 
event is applied, and each parallel sink writer must observe one schema epoch 
before any post-change record is accepted.
   3. Restore/replay behavior when an external JDBC DDL has succeeded but the 
corresponding checkpoint has not, including the idempotency mechanism for every 
supported mutation.
   4. SQL output semantics: a projected-away upstream column must not create a 
sink column, while a projected type or alias must produce an event matching the 
transform output rather than the upstream table.
   5. Recovery evidence: an E2E matrix with parallel sinks plus failures before 
apply, during apply, after apply/before checkpoint, and restore on both sides 
of the schema-change boundary.
   
   Once that contract is agreed, split the implementation so it can compose 
with #12284 and #11912 rather than duplicate either branch. No labels or 
assignments were changed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to