hesam-oxe opened a new pull request, #11077:
URL: https://github.com/apache/seatunnel/pull/11077
Prevents creating 200+ individual PaimonSinkWriter instances when
writing multiple sharded MySQL tables to a single Paimon target.
## Problem
When 200+ sharded source tables are routed to the same Paimon
destination, MultiTableSinkWriter creates one SinkWriter per logical
table. Each writer independently opens HiveCatalog and Hadoop
connections, causing initialization to stall (#10978).
## Fix
Added `sharedWriterCache` (ConcurrentHashMap) that maps destination
table IDs to a single shared SinkWriter. All sub-writer entries
targeting the same destination now reuse one underlying writer
instead of creating individual connections.
## Changes
- Added `sharedWriterCache` field for writer reuse
- Updated constructor to use `computeIfAbsent` for shared writers
- Updated `close()` to close shared writers once after all
synchronized blocks
- Updated `removeTableWriters()` to clean up cache entries
Closes #10978
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
