SEZ9 commented on issue #10203: URL: https://github.com/apache/seatunnel/issues/10203#issuecomment-6096357599
@ryanmeowy thanks for the detailed proposal and test plan, and for picking this up given there's been no linked PR for the slice @mzw111 described. Before you open anything, one concern about the Phase 1 attach point. The restore path in `IncrementalSource` already discovers the current tables from the source and computes `capturedTables - checkpointCapturedTables`, appending newly matched tables to the remaining snapshot work, and the MySQL factory already exposes `table_pattern` as the alternative to explicit `table_names`. So "discover newly matching tables on startup/restore" is effectively what runs today when a job is restored with a regex configured. Adding a default-off gate there would mostly change existing restore behavior rather than deliver what this issue asks for, which is picking up new tables without a restart. There is also unmerged work touching the per-table restore watermarks at exactly this boundary, so I'd rather not have a second PR altering table admission or configuration while that state/exactly-once path is still under review. Concrete asks: 1. @mzw111 — are you still planning to work on this? If not, please say so here so @ryanmeowy can take it. 2. @ryanmeowy — rather than the startup/restore gate, could you write up a design for the actually missing piece, live discovery while the binlog stream is running: trigger cadence (poll vs. DDL-event driven), enumerator ownership of newly admitted tables, what gets persisted in checkpoint state, split reassignment, behavior on checkpoint failure/recovery, duplicate-admission suppression, and the exact MySQL vs. OceanBase scope. 3. Please confirm in that design that existing `table_pattern` restore semantics stay backward compatible; your test items (c) and (d) are a good start for that. Once the design is agreed and the pending restore-watermark work lands, a PR against that design would be very welcome. <!-- streview-comment:1650 --> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
