Big +1, thanks this will be a very useful addition to Flink.
Best
Etienne
Le 07/12/2023 à 09:26, Hang Ruan a écrit :
+1 for contributing CDC Connectors to Apache Flink.
Best,
Hang
Yuxin Tan<tanyuxinw...@gmail.com> 于2023年12月7日周四 16:05写道：
Cool, +1 for contributing CDC Connectors to Apache Flink.
Best,
Yuxin
Jing Ge<j...@ververica.com.invalid> 于2023年12月7日周四 15:43写道：
Awesome! +1
Best regards,
Jing
On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin<snuyan...@gmail.com>
wrote:
thanks for working on this and driving it
+1
On Thu, Dec 7, 2023 at 7:26 AM Feng Jin<jinfeng1...@gmail.com> wrote:
This is incredibly exciting news, a big +1 for this.
Thank you for the fantastic work on Flink CDC. We have created
thousands
of
real-time integration jobs using Flink CDC connectors.
Best,
Feng
On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang <
gongzhongqi...@apache.org
wrote:
It's very exciting to hear the news.
+1 for adding CDC Connectors to Apache Flink !
Best,
Zhongqiang
Leonard Xu<xbjt...@gmail.com> 于2023年12月7日周四 11:25写道：
Dear Flink devs,
As you may have heard, we at Alibaba (Ververica) are planning to
donate
CDC Connectors for the Apache Flink project
*[1]* to the Apache Flink community.
CDC Connectors for Apache Flink comprise a collection of source
connectors designed specifically for Apache Flink. These connectors
*[2]*
enable the ingestion of changes from various databases using
Change
Data Capture (CDC), most of these CDC connectors are powered by
Debezium
*[3]*
. They support both the DataStream API and the Table/SQL API,
facilitating the reading of database snapshots and continuous
reading
of
transaction logs with exactly-once processing, even in the event of
failures.
Additionally, in the latest version 3.0, we have introduced many
long-awaited features. Starting from CDC version 3.0, we've built a
Streaming ELT Framework available for streaming data integration.
This
framework allows users to write their data synchronization logic
in a
simple YAML file, which will automatically be translated into a
Flink
DataStreaming job. It emphasizes optimizing the task submission
process
and
offers advanced functionalities such as whole database
synchronization,
merging sharded tables, and schema evolution
*[4]*.
I believe this initiative is a perfect match for both sides. For
the
Flink community, it presents an opportunity to enhance Flink's
competitive
advantage in streaming data integration, promoting the healthy
growth
and
prosperity of the Apache Flink ecosystem. For the CDC Connectors
project,
becoming a sub-project of Apache Flink means being part of a
neutral
open-source community, which can attract a more diverse pool of
contributors.
Please note that the aforementioned points represent only some of
our
motivations and vision for this donation. Specific future
operations
need
to be further discussed in this thread. For example, the
sub-project
name
after the donation; we hope to name it Flink-CDC
aiming to streaming data intergration through Apache Flink,
following the naming convention of Flink-ML; And this project is
managed
by a total of 8 maintainers, including 3 Flink PMC members and 1
Flink
Committer. The remaining 4 maintainers are also highly active
contributors
to the Flink community, donating this project to the Flink
community
implies that their permissions might be reduced. Therefore, we may
need
to
bring up this topic for further discussion within the Flink PMC.
Additionally, we need to discuss how to migrate existing users and
documents. We have a user group of nearly 10,000 people and a
multi-version
documentation site need to migrate. We also need to plan for the
migration
of CI/CD processes and other specifics.
While there are many intricate details that require
implementation,
we
are committed to progressing and finalizing this donation process.
Despite being Flink’s most active ecological project (as
evaluated
by
GitHub metrics), it also boasts a significant user base. However, I
believe
it's essential to commence discussions on future operations only
after
the
community reaches a consensus on whether they desire this donation.
Really looking forward to hear what you think!
Best,
Leonard (on behalf of the Flink CDC Connectors project
maintainers)
[1]https://github.com/ververica/flink-cdc-connectors
[2]
https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html
[3]https://debezium.io
[4]
https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
--
Best regards,
Sergey
