goutamadwant commented on issue #12565: URL: https://github.com/apache/seatunnel/issues/12565#issuecomment-6433551133
@DanielLeens I've opened #13006 for this as a focused replacement for #12567. The restart position after a restore stays as it is on dev, at the earliest offset of the restored splits. On top of that, the enumerator now keeps each restored table's own offset and passes it in the rebuilt incremental split. The exactly-once filter starts every old table from that offset instead of dropping it for having no snapshot info. The offset also stays in the reader checkpoint until the stream has read past it, so restoring again in the middle of the replay keeps it. Before and after, with MySQL, `exactly_once = true`: 1. Take a savepoint. 2. Add a third table. 3. Write 20 rows to an old table, some while the job is stopped and some while the new table is being snapshotted. 4. Restore. | | Old table in the sink | |---|---| | dev | 109 of 129 rows; those 20 rows never arrive | | with #13006 | 129 of 129, and an append-only copy of the sink (no primary key) shows no row twice | For `incremental.parallelism > 1`, unit tests cover two restored splits at different offsets regrouped into one: no missing rows, and no replay before a table's own offset. The e2e has to use a single incremental split, because restoring a MySQL savepoint with more than one is rejected. Older checkpoints still restore. @WXPNG, rechecking your old-table/new-table row counts against #13006 would be a useful second check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
