Hey Taher, Xianqian has replied your email, did you subscribe Flink dev mailing-list?
Best, Leonard > 2025年2月11日 16:52，Xiqian YU <kono....@outlook.com> 写道： > > Hi Taher, > > Since we’re creating a DataStream-based pipeline job with SQL Server CDC, > schema change events must be handled manually. A possible approach would be: > > 1) Enable schema change events with `.includeSchemaChanges(true)` option, so > DDL events will be parsed and encoded in `SourceRecord`s. > > 2) Write a customized `DebeziumDeserializationSchema` class and parse schema > change events. `MySqlEventDeserializer#deserializeSchemaChangeRecord` could > be used as a reference [1]. > > 3) Evolve sink schema of Paimon tables with `PaimonCatalog` manually. > `PaimonMetadataApplier` [2] is an existing schema evolving implementation > supporting a few frequently used schema change events. > > Also, CDC Pipeline framework [3] has provided a fully-automatic schema > sensing and evolving solution, but unfortunately Microsoft SQL Server source > is not supported yet until we close #3445 [4] or #3507 [5]. > > [1] > https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-mysql/src/main/java/org/apache/flink/cdc/connectors/mysql/source/MySqlEventDeserializer.java > [2] > https://github.com/apache/flink-cdc/blob/master/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-pipeline-connector-paimon/src/main/java/org/apache/flink/cdc/connectors/paimon/sink/PaimonMetadataApplier.java > [3] > https://nightlies.apache.org/flink/flink-cdc-docs-release-3.3/docs/core-concept/data-pipeline/ > [4] https://github.com/apache/flink-cdc/pull/3445 > [5] https://github.com/apache/flink-cdc/pull/3507 > > Best Regards, > Xiqian > > Taher Koitawala <taher...@gmail.com> 於 2025年2月11日 15:59 寫道： > > Hi Devs, > As a POC we are trying to create a steaming pipeline from MSSQL cdc > to Paimon: > > To do this we are doing > 1. msSql server cdc operator > 2. Transform operator > 3. paimon sink > > We have written the cdc connector with is a JsonDebeziumDeserialisedSchema > String > > I wish to write this paimon in a table format with same columns as source. > > As far as i know paimon automatically handles schema updates like new field > additions. > > Please can someone point me on how to write this stream efficiently to > paimon table with schema updates? > > For now i have SouceFunction<String> > > Which is the record mentioned above! > > Regards, > Taher Koitawala >
