DanielLeens commented on PR #11912: URL: https://github.com/apache/seatunnel/pull/11912#issuecomment-6466886383
Thanks @nzw921rx for the detailed P1 on `d3a8a348ddb0`. I re-read `SchemaOperator` and `SchemaRefreshBarrierOperator` on this head against your scenario (static read only, nothing run locally), and I agree the finding is real. - `SchemaOperator#notifyCheckpointComplete` releases the buffered rows through `completeNextPendingSchemaChange` as soon as `checkpointId > schemaChangeDispatchedCheckpointId`. A larger ID only says the checkpoint completed later; it does not say its barrier passed this operator after the control record was sent. With `execution.max-concurrent-checkpoints > 1`, checkpoint N+1 can already have been snapshotted before the callback for N dispatches the DDL, so its completion can release the rows too early. - On the sink side, `SchemaRefreshBarrierOperator#processElement1` forwards a data row unchanged when `findSchemaSnapshot` returns null, and the refresh arrives separately on input 2 (`processElement2`). The comment saying the marker and the row share one edge only holds for the locally emitted marker, not for a delayed control input. So a released row can reach the writer with the old schema, and a JDBC writer would silently omit the new column, as in your SQLite probe. - With the default of one concurrent checkpoint the window should not open, which is why the existing tests do not catch it. That narrows the exposure but does not make it safe. This needs a code change, so I am treating it as a blocking item on my side and I have not marked it resolved. @dybyte, please make row release depend on a checkpoint whose barrier is known to follow the dispatched control record, or on an explicit ack from the downstream refresh, and add a regression test with an in-flight checkpoint plus delayed refresh delivery that asserts the new column keeps its value. I will re-review as soon as a new commit lands. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
