qidian99 opened a new issue, #1181: URL: https://github.com/apache/incubator-paimon/issues/1181
### Search before asking - [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar. ### Motivation After CdcSyncDatabaseAction job start (for now we only sync a single database), if new table is created in MySQL, then action should sync the new mysql table to paimon. Specifically, 1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix. 2. CDC changelog of the newly added table should be successfully written to Paimon 3. Support multiple newly added tables. ### Solution The draft design is as follows: 1. The new Paimon table is created upon receiving the create table DDL event in CDC source. Need to respect prefix/suffix. Therefore we need to parse create table statement and utilize Paimon catalog to create the physical table 2. CDC changelog of the newly added table should be successfully written to Paimon So we need another type of Paimon sink designed to write changelog from newly added tables to their corresponding Paimon tables. 3. Support multiple newly added tables. Specifically, the new Paimon sink needs to write the data from newly added tables discovered during runtime by managing multiple Paimon tables at the same time. ### Anything else? To accomplish this feature, it's better to go by several steps: 1. PoC: implement a first draft to sync newly added table. 2. Implement a multiplexed Paimon sink to support multi-table write 3. Implement a multiplexed Paimon committer to support multi-table commit 4. End-to-end test against newly added table scenario ### Are you willing to submit a PR? - [X] I'm willing to submit a PR! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
