DanielLeens commented on issue #10978: URL: https://github.com/apache/seatunnel/issues/10978#issuecomment-4581048943
I traced the current multi-table sink path, and this looks closer to a real framework limitation / enhancement item than a simple usage problem. Today the Flink multi-table sink path still creates one sub-writer per logical table. In the current code, `MultiTableSinkWriter` is initialized with the full `sinkWriters.size()`, and resource sharing happens after those table-scoped writer instances already exist. So when 200+ sharded source tables are routed into one Paimon target, you can still end up paying for 200+ sink-side writer contexts during initialization. That matches the symptom you described much better than a pure config mistake. At the moment there is no existing config option that collapses all of those logical source tables into a single sink writer. So I think this should stay open as an enhancement in the multi-table sink layer. If you can add two more pieces of information, we can narrow the bottleneck further: 1. whether those 200+ tables come from one source block with table expansion or from many separately generated source tables; 2. the log line or stack trace showing where initialization stalls (catalog load, writer open, commit preparation, etc.). We have labeled this as `help wanted` as well, since contributions in the multi-table sink optimization path would be very valuable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
