nzw921rx commented on PR #11077: URL: https://github.com/apache/seatunnel/pull/11077#issuecomment-6845260477
I ran a local validation on the PR base (`9cd7defe`), without this PR’s changes, using 200 same-schema FakeSource tables, one Sink subtask, and `multi_table_sink_replica = 1`. | Pipeline | Paimon Sink instances | Paimon Writers | |---|---:|---:| | Source → Paimon | 200 | 200 | | Source → TableMerge → Paimon | 1 | 1 | With `TableMerge`, 2,000 records were successfully written, committed, and read back from Paimon. All 200 source tables were represented, with no missing or duplicate records. This test used Paimon’s local filesystem catalog. It does not reproduce the original MySQL + Hive Metastore environment, but it confirms that `TableMerge` eliminates the per-source-table Sink/Writer creation in this same-schema case. Given the substantial changes to the core multi-table sink lifecycle, could you first verify whether **Source → TableMerge → Paimon** resolves the original issue? If `TableMerge` cannot cover a specific requirement, please explain that limitation. We could then consider extending it or introducing a dedicated transform to handle the merging before the Sink, keeping the change within the Transform layer where possible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
