DanielLeens commented on issue #10873: URL: https://github.com/apache/seatunnel/issues/10873#issuecomment-4426577976
Thanks for writing this up so concretely, and for opening draft PR #10874 right away. I agree this should be treated as a larger design / lifecycle topic rather than just "add one more event". In the current codebase, the table-complete path is not really a first-class contract yet: - `SourceReader.Context#signalNoMoreElement()` is reader-global, not table-scoped - the engine data path has explicit handling for `Barrier` and `SchemaChangeEvent`, but not a generic table-lifecycle control event - `MultiTableSinkWriter` can route rows and apply schema changes per table, but early per-table close still needs explicit parallel-completion semantics to avoid reclaiming a table too early So the overall direction makes sense, but I would strongly suggest keeping the first merge target intentionally narrow: 1. one bounded multi-table source path first, for example JDBC 2. only multi-table sinks that explicitly opt in to early table close behavior 3. explicit non-goals for unbounded sources and other engine / translation paths in the first PR 4. clear notes about rename mapping, transform passthrough, and checkpoint / ordering guarantees That scope will make the change much easier to review and much safer from a compatibility perspective. If PR #10874 documents those boundaries clearly and proves the upstream-reader counting logic with focused tests, this issue looks worth tracking as a design / improvement thread while implementation continues there. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
