davidzollo opened a new issue, #10873: URL: https://github.com/apache/seatunnel/issues/10873
### Search before creating I checked the current Apache SeaTunnel issues and pull requests on May 11, 2026 and did not find an existing issue/PR that covers this bounded multi-table close-table lifecycle gap end to end. ### Background In bounded multi-table jobs, SeaTunnel can know that one logical table has been fully consumed before the whole pipeline finishes. However, the current open-source pipeline does not propagate a table-complete signal to downstream multi-table sinks. As a result, table-scoped sink resources stay alive until the whole task closes, for example: - per-table sink writers - table-scoped buffers - schema caches - connection / resource-manager entries owned by multi-table sinks This becomes more visible for sinks that keep table-level state for many tables in one job. ### Current problem There are two related gaps today: 1. There is no generic `CloseTableEvent` path from bounded sources through engine / transforms into sinks. 2. In parallel jobs, a single "table finished" event is not enough. If one upstream reader emits a close signal before other readers finish delivering their last records for the same table, the sink may reclaim the table too early. So the missing behavior is not only "emit one event", but "emit a safe per-table completion signal that works with multiple upstream readers". ### Expected behavior For bounded multi-table jobs: - participating upstream readers should emit close-table progress after they finish producing records for a table - downstream transforms should preserve that control event - multi-table sinks should only close a table after they have received all expected upstream close-table signals for that table This lets sinks reclaim table-scoped resources early without breaking record ordering in parallel pipelines. ### Proposed direction Introduce a table lifecycle control path for bounded multi-table jobs: - add a generic close-table event in `seatunnel-api` - allow engine source enumerators to send reader events when table-level progress completes - propagate close-table events through transform and sink flow lifecycles - let `MultiTableSinkWriter` count upstream close-table signals and close only when all participating readers have reported completion - wire one concrete bounded multi-table source path first, for example JDBC ### Impact This improves memory/resource usage for large bounded multi-table jobs and gives sinks a safe hook to release table-scoped resources before the full job ends. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
