A big +1 for this proposal. Thanks Leonard and the Flink CDC community. While there are many further details to be discussed, I believe this proposal is aligned with the long-term interests for both communities.
Best, Xintong On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb <decordea...@gmail.com> wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li <jingsongl...@gmail.com> wrote: > > > Wow, Cool, Nice > > > > CDC is playing an increasingly important role. > > > > +1 > > > > Best, > > Jingsong > > > > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu <xbjt...@gmail.com> wrote: > > > > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > > CDC Connectors for the Apache Flink project[1] to the Apache Flink > > community. > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > connectors designed specifically for Apache Flink. These connectors[2] > > enable the ingestion of changes from various databases using Change Data > > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > > They support both the DataStream API and the Table/SQL API, facilitating > > the reading of database snapshots and continuous reading of transaction > > logs with exactly-once processing, even in the event of failures. > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process > and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution[4]. > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > > Flink community, it presents an opportunity to enhance Flink's > competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC aiming to streaming data > > intergration through Apache Flink, following the naming convention of > > Flink-ML; And this project is managed by a total of 8 maintainers, > > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > maintainers are also highly active contributors to the Flink community, > > donating this project to the Flink community implies that their > permissions > > might be reduced. Therefore, we may need to bring up this topic for > further > > discussion within the Flink PMC. Additionally, we need to discuss how to > > migrate existing users and documents. We have a user group of nearly > 10,000 > > people and a multi-version documentation site need to migrate. We also > need > > to plan for the migration of CI/CD processes and other specifics. > > > > > > > > > While there are many intricate details that require implementation, we > > are committed to progressing and finalizing this donation process. > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I > believe > > it's essential to commence discussions on future operations only after > the > > community reaches a consensus on whether they desire this donation. > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > Best, > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > [2] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > [3] https://debezium.io > > > [4] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > >
