+1 for adding this to Apache Flink! I think this can further extend the ability of Apache Flink and a lot of users would be interested to try this out.
Best, Jark On Thu, 7 Dec 2023 at 12:06, Samrat Deb <decordea...@gmail.com> wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <jingsongl...@gmail.com> wrote: > >> Wow, Cool, Nice >> >> CDC is playing an increasingly important role. >> >> +1 >> >> Best, >> Jingsong >> >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xbjt...@gmail.com> wrote: >> > >> > Dear Flink devs, >> > >> > As you may have heard, we at Alibaba (Ververica) are planning to donate >> CDC Connectors for the Apache Flink project[1] to the Apache Flink >> community. >> > >> > CDC Connectors for Apache Flink comprise a collection of source >> connectors designed specifically for Apache Flink. These connectors[2] >> enable the ingestion of changes from various databases using Change Data >> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. >> They support both the DataStream API and the Table/SQL API, facilitating >> the reading of database snapshots and continuous reading of transaction >> logs with exactly-once processing, even in the event of failures. >> > >> > >> > Additionally, in the latest version 3.0, we have introduced many >> long-awaited features. Starting from CDC version 3.0, we've built a >> Streaming ELT Framework available for streaming data integration. This >> framework allows users to write their data synchronization logic in a >> simple YAML file, which will automatically be translated into a Flink >> DataStreaming job. It emphasizes optimizing the task submission process and >> offers advanced functionalities such as whole database synchronization, >> merging sharded tables, and schema evolution[4]. >> > >> > >> > I believe this initiative is a perfect match for both sides. For the >> Flink community, it presents an opportunity to enhance Flink's competitive >> advantage in streaming data integration, promoting the healthy growth and >> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, >> becoming a sub-project of Apache Flink means being part of a neutral >> open-source community, which can attract a more diverse pool of >> contributors. >> > >> > Please note that the aforementioned points represent only some of our >> motivations and vision for this donation. Specific future operations need >> to be further discussed in this thread. For example, the sub-project name >> after the donation; we hope to name it Flink-CDC aiming to streaming data >> intergration through Apache Flink, following the naming convention of >> Flink-ML; And this project is managed by a total of 8 maintainers, >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 >> maintainers are also highly active contributors to the Flink community, >> donating this project to the Flink community implies that their permissions >> might be reduced. Therefore, we may need to bring up this topic for further >> discussion within the Flink PMC. Additionally, we need to discuss how to >> migrate existing users and documents. We have a user group of nearly 10,000 >> people and a multi-version documentation site need to migrate. We also need >> to plan for the migration of CI/CD processes and other specifics. >> > >> > >> > While there are many intricate details that require implementation, we >> are committed to progressing and finalizing this donation process. >> > >> > >> > Despite being Flink’s most active ecological project (as evaluated by >> GitHub metrics), it also boasts a significant user base. However, I believe >> it's essential to commence discussions on future operations only after the >> community reaches a consensus on whether they desire this donation. >> > >> > >> > Really looking forward to hear what you think! >> > >> > >> > Best, >> > Leonard (on behalf of the Flink CDC Connectors project maintainers) >> > >> > [1] https://github.com/ververica/flink-cdc-connectors >> > [2] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html >> > [3] https://debezium.io >> > [4] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >> >
