Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Awesome! Big +1. Regards, Nicholas Jiang On 2023/12/07 03:24:59 Leonard Xu wrote: > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Good News. +1
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Hey, Zarzycki Thanks for your reply and recognization ! > As I understand you're thinking with Flink CDC 3.0 to make the project > broader than CDC, as more "Streaming Integration" framework. This will > resemble Kafka Connect, what about renaming the project to something like > "Flink Connect"? :-) > It would be amazing to have such a tool, "swiss army knife" for data > integration, based on Flink. Flink Community has a lot of connectors thus the Flink Connect name may not suitable. Although we offers end-to-end streaming integration capability , we still want to use the current name Flink CDC as we want to do best in database CDC area and users has accepted/known the name well. While the project name holds some significance, I believe its true importance lies in its ability to effectively address problems and deliver substantial value to the community users. Best, Leonard > > > On Tue, Dec 12, 2023 at 3:01 PM Lincoln Lee wrote: > >> +1 >> Thanks Leonard and all CDC contributors for the great work! >> >> Best, >> Lincoln Lee >> >> >> Dian Fu 于2023年12月12日周二 19:11写道： >> >>> Big +1 for this! Thanks for driving this effort! >>> >>> Regards, >>> Dian >>> >>> On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu wrote: Hey all, I want to extend my gratitude for the positive feedback we've received, >>> particularly from voices in other communities. It is an honor for the >> Flink >>> CDC community to have received your recognition. Moving forward, I will collaborate with the Flink CDC community >>> maintainers to prepare a FLIP (Flink Improvement Proposal) document that >>> will outline specific plans and implementation details. Thank you for your support, and we look forward to contributing to >>> Apache Flink community. Best, Leonard > 2023年12月11日 下午4:12，yh z 写道： > > Excited to hear the news, > +1 > > Best, > Yunhong (Swuferhong) > > liu ron 于2023年12月11日周一 14:05写道： > >> +1 >> >> Best, >> Ron >> >> Yunqing Mo 于2023年12月11日周一 12:01写道： >> >>> So cool, Big +1 for this exciting work. >>> >>> On 2023/12/07 03:24:59 Leonard Xu wrote: Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to >>> donate >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink >>> community. CDC Connectors for Apache Flink comprise a collection of source >>> connectors designed specifically for Apache Flink. These >>> connectors[2] >>> enable the ingestion of changes from various databases using Change >>> Data >>> Capture (CDC), most of these CDC connectors are powered by >>> Debezium[3]. >>> They support both the DataStream API and the Table/SQL API, >>> facilitating >>> the reading of database snapshots and continuous reading of >>> transaction >>> logs with exactly-once processing, even in the event of failures. Additionally, in the latest version 3.0, we have introduced many >>> long-awaited features. Starting from CDC version 3.0, we've built a >>> Streaming ELT Framework available for streaming data integration. >>> This >>> framework allows users to write their data synchronization logic >> in a >>> simple YAML file, which will automatically be translated into a >> Flink >>> DataStreaming job. It emphasizes optimizing the task submission >>> process >> and >>> offers advanced functionalities such as whole database >>> synchronization, >>> merging sharded tables, and schema evolution[4]. I believe this initiative is a perfect match for both sides. For >> the >>> Flink community, it presents an opportunity to enhance Flink's >> competitive >>> advantage in streaming data integration, promoting the healthy >>> growth and >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors >>> project, >>> becoming a sub-project of Apache Flink means being part of a >> neutral >>> open-source community, which can attract a more diverse pool of >>> contributors. Please note that the aforementioned points represent only some of >>> our >>> motivations and vision for this donation. Specific future >> operations >>> need >>> to be further discussed in this thread. For example, the >> sub-project >>> name >>> after the donation; we hope to name it Flink-CDC aiming to >> streaming >>> data >>> intergration through Apache Flink, following the naming convention >> of >>> Flink-ML; And this project is managed by a total of 8 maintainers, >>> including 3 Flink PMC members and 1 Flink Committer. The remaining >> 4 >>> maintainers are also highly active contributors to the Flink >>> community, >>> donating this project to the Flink community implies that their >> permissions >>> might
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Hello, I'm super excited that Flink CDC will join the Flink project! It will be so much easier to bring it into Flink-based projects we run. As I understand you're thinking with Flink CDC 3.0 to make the project broader than CDC, as more "Streaming Integration" framework. This will resemble Kafka Connect, what about renaming the project to something like "Flink Connect"? :-) It would be amazing to have such a tool, "swiss army knife" for data integration, based on Flink. Best Krzysztof On Tue, Dec 12, 2023 at 3:01 PM Lincoln Lee wrote: > +1 > Thanks Leonard and all CDC contributors for the great work! > > Best, > Lincoln Lee > > > Dian Fu 于2023年12月12日周二 19:11写道： > > > Big +1 for this! Thanks for driving this effort! > > > > Regards, > > Dian > > > > On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu wrote: > > > > > > Hey all, > > > > > > I want to extend my gratitude for the positive feedback we've received, > > particularly from voices in other communities. It is an honor for the > Flink > > CDC community to have received your recognition. > > > > > > Moving forward, I will collaborate with the Flink CDC community > > maintainers to prepare a FLIP (Flink Improvement Proposal) document that > > will outline specific plans and implementation details. > > > > > > Thank you for your support, and we look forward to contributing to > > Apache Flink community. > > > > > > Best, > > > Leonard > > > > > > > 2023年12月11日 下午4:12，yh z 写道： > > > > > > > > Excited to hear the news, > > > > +1 > > > > > > > > Best, > > > > Yunhong (Swuferhong) > > > > > > > > liu ron 于2023年12月11日周一 14:05写道： > > > > > > > >> +1 > > > >> > > > >> Best, > > > >> Ron > > > >> > > > >> Yunqing Mo 于2023年12月11日周一 12:01写道： > > > >> > > > >>> So cool, Big +1 for this exciting work. > > > >>> > > > >>> On 2023/12/07 03:24:59 Leonard Xu wrote: > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to > > donate > > > >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink > > > >>> community. > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > > >>> connectors designed specifically for Apache Flink. These > > connectors[2] > > > >>> enable the ingestion of changes from various databases using Change > > Data > > > >>> Capture (CDC), most of these CDC connectors are powered by > > Debezium[3]. > > > >>> They support both the DataStream API and the Table/SQL API, > > facilitating > > > >>> the reading of database snapshots and continuous reading of > > transaction > > > >>> logs with exactly-once processing, even in the event of failures. > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > > >>> long-awaited features. Starting from CDC version 3.0, we've built a > > > >>> Streaming ELT Framework available for streaming data integration. > > This > > > >>> framework allows users to write their data synchronization logic > in a > > > >>> simple YAML file, which will automatically be translated into a > Flink > > > >>> DataStreaming job. It emphasizes optimizing the task submission > > process > > > >> and > > > >>> offers advanced functionalities such as whole database > > synchronization, > > > >>> merging sharded tables, and schema evolution[4]. > > > > > > > > > I believe this initiative is a perfect match for both sides. For > the > > > >>> Flink community, it presents an opportunity to enhance Flink's > > > >> competitive > > > >>> advantage in streaming data integration, promoting the healthy > > growth and > > > >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors > > project, > > > >>> becoming a sub-project of Apache Flink means being part of a > neutral > > > >>> open-source community, which can attract a more diverse pool of > > > >>> contributors. > > > > > > Please note that the aforementioned points represent only some of > > our > > > >>> motivations and vision for this donation. Specific future > operations > > need > > > >>> to be further discussed in this thread. For example, the > sub-project > > name > > > >>> after the donation; we hope to name it Flink-CDC aiming to > streaming > > data > > > >>> intergration through Apache Flink, following the naming convention > of > > > >>> Flink-ML; And this project is managed by a total of 8 maintainers, > > > >>> including 3 Flink PMC members and 1 Flink Committer. The remaining > 4 > > > >>> maintainers are also highly active contributors to the Flink > > community, > > > >>> donating this project to the Flink community implies that their > > > >> permissions > > > >>> might be reduced. Therefore, we may need to bring up this topic for > > > >> further > > > >>> discussion within the Flink PMC. Additionally, we need to discuss > > how to > > > >>> migrate existing users and documents. We have a user group of > nearly > > > >> 10,000 > > > >>> people and a multi-version documentation site
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 Thanks Leonard and all CDC contributors for the great work! Best, Lincoln Lee Dian Fu 于2023年12月12日周二 19:11写道： > Big +1 for this! Thanks for driving this effort! > > Regards, > Dian > > On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu wrote: > > > > Hey all, > > > > I want to extend my gratitude for the positive feedback we've received, > particularly from voices in other communities. It is an honor for the Flink > CDC community to have received your recognition. > > > > Moving forward, I will collaborate with the Flink CDC community > maintainers to prepare a FLIP (Flink Improvement Proposal) document that > will outline specific plans and implementation details. > > > > Thank you for your support, and we look forward to contributing to > Apache Flink community. > > > > Best, > > Leonard > > > > > 2023年12月11日 下午4:12，yh z 写道： > > > > > > Excited to hear the news, > > > +1 > > > > > > Best, > > > Yunhong (Swuferhong) > > > > > > liu ron 于2023年12月11日周一 14:05写道： > > > > > >> +1 > > >> > > >> Best, > > >> Ron > > >> > > >> Yunqing Mo 于2023年12月11日周一 12:01写道： > > >> > > >>> So cool, Big +1 for this exciting work. > > >>> > > >>> On 2023/12/07 03:24:59 Leonard Xu wrote: > > Dear Flink devs, > > > > As you may have heard, we at Alibaba (Ververica) are planning to > donate > > >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink > > >>> community. > > > > CDC Connectors for Apache Flink comprise a collection of source > > >>> connectors designed specifically for Apache Flink. These > connectors[2] > > >>> enable the ingestion of changes from various databases using Change > Data > > >>> Capture (CDC), most of these CDC connectors are powered by > Debezium[3]. > > >>> They support both the DataStream API and the Table/SQL API, > facilitating > > >>> the reading of database snapshots and continuous reading of > transaction > > >>> logs with exactly-once processing, even in the event of failures. > > > > > > Additionally, in the latest version 3.0, we have introduced many > > >>> long-awaited features. Starting from CDC version 3.0, we've built a > > >>> Streaming ELT Framework available for streaming data integration. > This > > >>> framework allows users to write their data synchronization logic in a > > >>> simple YAML file, which will automatically be translated into a Flink > > >>> DataStreaming job. It emphasizes optimizing the task submission > process > > >> and > > >>> offers advanced functionalities such as whole database > synchronization, > > >>> merging sharded tables, and schema evolution[4]. > > > > > > I believe this initiative is a perfect match for both sides. For the > > >>> Flink community, it presents an opportunity to enhance Flink's > > >> competitive > > >>> advantage in streaming data integration, promoting the healthy > growth and > > >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors > project, > > >>> becoming a sub-project of Apache Flink means being part of a neutral > > >>> open-source community, which can attract a more diverse pool of > > >>> contributors. > > > > Please note that the aforementioned points represent only some of > our > > >>> motivations and vision for this donation. Specific future operations > need > > >>> to be further discussed in this thread. For example, the sub-project > name > > >>> after the donation; we hope to name it Flink-CDC aiming to streaming > data > > >>> intergration through Apache Flink, following the naming convention of > > >>> Flink-ML; And this project is managed by a total of 8 maintainers, > > >>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > >>> maintainers are also highly active contributors to the Flink > community, > > >>> donating this project to the Flink community implies that their > > >> permissions > > >>> might be reduced. Therefore, we may need to bring up this topic for > > >> further > > >>> discussion within the Flink PMC. Additionally, we need to discuss > how to > > >>> migrate existing users and documents. We have a user group of nearly > > >> 10,000 > > >>> people and a multi-version documentation site need to migrate. We > also > > >> need > > >>> to plan for the migration of CI/CD processes and other specifics. > > > > > > While there are many intricate details that require implementation, > we > > >>> are committed to progressing and finalizing this donation process. > > > > > > Despite being Flink’s most active ecological project (as evaluated > by > > >>> GitHub metrics), it also boasts a significant user base. However, I > > >> believe > > >>> it's essential to commence discussions on future operations only > after > > >> the > > >>> community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > Best, > > Leonard (on behalf of the Flink CDC
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Big +1 for this! Thanks for driving this effort! Regards, Dian On Mon, Dec 11, 2023 at 4:48 PM Leonard Xu wrote: > > Hey all, > > I want to extend my gratitude for the positive feedback we've received, > particularly from voices in other communities. It is an honor for the Flink > CDC community to have received your recognition. > > Moving forward, I will collaborate with the Flink CDC community maintainers > to prepare a FLIP (Flink Improvement Proposal) document that will outline > specific plans and implementation details. > > Thank you for your support, and we look forward to contributing to Apache > Flink community. > > Best, > Leonard > > > 2023年12月11日 下午4:12，yh z 写道： > > > > Excited to hear the news, > > +1 > > > > Best, > > Yunhong (Swuferhong) > > > > liu ron 于2023年12月11日周一 14:05写道： > > > >> +1 > >> > >> Best, > >> Ron > >> > >> Yunqing Mo 于2023年12月11日周一 12:01写道： > >> > >>> So cool, Big +1 for this exciting work. > >>> > >>> On 2023/12/07 03:24:59 Leonard Xu wrote: > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate > >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink > >>> community. > > CDC Connectors for Apache Flink comprise a collection of source > >>> connectors designed specifically for Apache Flink. These connectors[2] > >>> enable the ingestion of changes from various databases using Change Data > >>> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > >>> They support both the DataStream API and the Table/SQL API, facilitating > >>> the reading of database snapshots and continuous reading of transaction > >>> logs with exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many > >>> long-awaited features. Starting from CDC version 3.0, we've built a > >>> Streaming ELT Framework available for streaming data integration. This > >>> framework allows users to write their data synchronization logic in a > >>> simple YAML file, which will automatically be translated into a Flink > >>> DataStreaming job. It emphasizes optimizing the task submission process > >> and > >>> offers advanced functionalities such as whole database synchronization, > >>> merging sharded tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the > >>> Flink community, it presents an opportunity to enhance Flink's > >> competitive > >>> advantage in streaming data integration, promoting the healthy growth and > >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > >>> becoming a sub-project of Apache Flink means being part of a neutral > >>> open-source community, which can attract a more diverse pool of > >>> contributors. > > Please note that the aforementioned points represent only some of our > >>> motivations and vision for this donation. Specific future operations need > >>> to be further discussed in this thread. For example, the sub-project name > >>> after the donation; we hope to name it Flink-CDC aiming to streaming data > >>> intergration through Apache Flink, following the naming convention of > >>> Flink-ML; And this project is managed by a total of 8 maintainers, > >>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > >>> maintainers are also highly active contributors to the Flink community, > >>> donating this project to the Flink community implies that their > >> permissions > >>> might be reduced. Therefore, we may need to bring up this topic for > >> further > >>> discussion within the Flink PMC. Additionally, we need to discuss how to > >>> migrate existing users and documents. We have a user group of nearly > >> 10,000 > >>> people and a multi-version documentation site need to migrate. We also > >> need > >>> to plan for the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we > >>> are committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by > >>> GitHub metrics), it also boasts a significant user base. However, I > >> believe > >>> it's essential to commence discussions on future operations only after > >> the > >>> community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > >>> > >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > >>> > >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > >>> > >> >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Hey all, I want to extend my gratitude for the positive feedback we've received, particularly from voices in other communities. It is an honor for the Flink CDC community to have received your recognition. Moving forward, I will collaborate with the Flink CDC community maintainers to prepare a FLIP (Flink Improvement Proposal) document that will outline specific plans and implementation details. Thank you for your support, and we look forward to contributing to Apache Flink community. Best, Leonard > 2023年12月11日 下午4:12，yh z 写道： > > Excited to hear the news, > +1 > > Best, > Yunhong (Swuferhong) > > liu ron 于2023年12月11日周一 14:05写道： > >> +1 >> >> Best, >> Ron >> >> Yunqing Mo 于2023年12月11日周一 12:01写道： >> >>> So cool, Big +1 for this exciting work. >>> >>> On 2023/12/07 03:24:59 Leonard Xu wrote: Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to donate >>> CDC Connectors for the Apache Flink project[1] to the Apache Flink >>> community. CDC Connectors for Apache Flink comprise a collection of source >>> connectors designed specifically for Apache Flink. These connectors[2] >>> enable the ingestion of changes from various databases using Change Data >>> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. >>> They support both the DataStream API and the Table/SQL API, facilitating >>> the reading of database snapshots and continuous reading of transaction >>> logs with exactly-once processing, even in the event of failures. Additionally, in the latest version 3.0, we have introduced many >>> long-awaited features. Starting from CDC version 3.0, we've built a >>> Streaming ELT Framework available for streaming data integration. This >>> framework allows users to write their data synchronization logic in a >>> simple YAML file, which will automatically be translated into a Flink >>> DataStreaming job. It emphasizes optimizing the task submission process >> and >>> offers advanced functionalities such as whole database synchronization, >>> merging sharded tables, and schema evolution[4]. I believe this initiative is a perfect match for both sides. For the >>> Flink community, it presents an opportunity to enhance Flink's >> competitive >>> advantage in streaming data integration, promoting the healthy growth and >>> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, >>> becoming a sub-project of Apache Flink means being part of a neutral >>> open-source community, which can attract a more diverse pool of >>> contributors. Please note that the aforementioned points represent only some of our >>> motivations and vision for this donation. Specific future operations need >>> to be further discussed in this thread. For example, the sub-project name >>> after the donation; we hope to name it Flink-CDC aiming to streaming data >>> intergration through Apache Flink, following the naming convention of >>> Flink-ML; And this project is managed by a total of 8 maintainers, >>> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 >>> maintainers are also highly active contributors to the Flink community, >>> donating this project to the Flink community implies that their >> permissions >>> might be reduced. Therefore, we may need to bring up this topic for >> further >>> discussion within the Flink PMC. Additionally, we need to discuss how to >>> migrate existing users and documents. We have a user group of nearly >> 10,000 >>> people and a multi-version documentation site need to migrate. We also >> need >>> to plan for the migration of CI/CD processes and other specifics. While there are many intricate details that require implementation, we >>> are committed to progressing and finalizing this donation process. Despite being Flink’s most active ecological project (as evaluated by >>> GitHub metrics), it also boasts a significant user base. However, I >> believe >>> it's essential to commence discussions on future operations only after >> the >>> community reaches a consensus on whether they desire this donation. Really looking forward to hear what you think! Best, Leonard (on behalf of the Flink CDC Connectors project maintainers) [1] https://github.com/ververica/flink-cdc-connectors [2] >>> >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html [3] https://debezium.io [4] >>> >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >>> >>
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Excited to hear the news, +1 Best, Yunhong (Swuferhong) liu ron 于2023年12月11日周一 14:05写道： > +1 > > Best, > Ron > > Yunqing Mo 于2023年12月11日周一 12:01写道： > > > So cool, Big +1 for this exciting work. > > > > On 2023/12/07 03:24:59 Leonard Xu wrote: > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > > CDC Connectors for the Apache Flink project[1] to the Apache Flink > > community. > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > connectors designed specifically for Apache Flink. These connectors[2] > > enable the ingestion of changes from various databases using Change Data > > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > > They support both the DataStream API and the Table/SQL API, facilitating > > the reading of database snapshots and continuous reading of transaction > > logs with exactly-once processing, even in the event of failures. > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process > and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution[4]. > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > > Flink community, it presents an opportunity to enhance Flink's > competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC aiming to streaming data > > intergration through Apache Flink, following the naming convention of > > Flink-ML; And this project is managed by a total of 8 maintainers, > > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > maintainers are also highly active contributors to the Flink community, > > donating this project to the Flink community implies that their > permissions > > might be reduced. Therefore, we may need to bring up this topic for > further > > discussion within the Flink PMC. Additionally, we need to discuss how to > > migrate existing users and documents. We have a user group of nearly > 10,000 > > people and a multi-version documentation site need to migrate. We also > need > > to plan for the migration of CI/CD processes and other specifics. > > > > > > > > > While there are many intricate details that require implementation, we > > are committed to progressing and finalizing this donation process. > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I > believe > > it's essential to commence discussions on future operations only after > the > > community reaches a consensus on whether they desire this donation. > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > Best, > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > [2] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > [3] https://debezium.io > > > [4] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 Best, Ron Yunqing Mo 于2023年12月11日周一 12:01写道： > So cool, Big +1 for this exciting work. > > On 2023/12/07 03:24:59 Leonard Xu wrote: > > Dear Flink devs, > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > CDC Connectors for the Apache Flink project[1] to the Apache Flink > community. > > > > CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These connectors[2] > enable the ingestion of changes from various databases using Change Data > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > They support both the DataStream API and the Table/SQL API, facilitating > the reading of database snapshots and continuous reading of transaction > logs with exactly-once processing, even in the event of failures. > > > > > > Additionally, in the latest version 3.0, we have introduced many > long-awaited features. Starting from CDC version 3.0, we've built a > Streaming ELT Framework available for streaming data integration. This > framework allows users to write their data synchronization logic in a > simple YAML file, which will automatically be translated into a Flink > DataStreaming job. It emphasizes optimizing the task submission process and > offers advanced functionalities such as whole database synchronization, > merging sharded tables, and schema evolution[4]. > > > > > > I believe this initiative is a perfect match for both sides. For the > Flink community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of > contributors. > > > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need > to be further discussed in this thread. For example, the sub-project name > after the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > maintainers are also highly active contributors to the Flink community, > donating this project to the Flink community implies that their permissions > might be reduced. Therefore, we may need to bring up this topic for further > discussion within the Flink PMC. Additionally, we need to discuss how to > migrate existing users and documents. We have a user group of nearly 10,000 > people and a multi-version documentation site need to migrate. We also need > to plan for the migration of CI/CD processes and other specifics. > > > > > > While there are many intricate details that require implementation, we > are committed to progressing and finalizing this donation process. > > > > > > Despite being Flink’s most active ecological project (as evaluated by > GitHub metrics), it also boasts a significant user base. However, I believe > it's essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > Best, > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > [1] https://github.com/ververica/flink-cdc-connectors > > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > [3] https://debezium.io > > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
So cool, Big +1 for this exciting work. On 2023/12/07 03:24:59 Leonard Xu wrote: > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Awesome, +1 Brs, di.wu > 2023年12月7日 上午11:24，Leonard Xu 写道： > > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 Great, cdc is very important for real-time data integration, and this project is also widely used, I am optimistic about this project Jiafeng.Zhang / 张家锋 Email: jiafengzh...@apache.org
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 On 2023/12/11 02:33:24 Xin Gong wrote: > > good news. > > +1 > > Best, > gongxin > On 2023/12/07 03:24:59 Leonard Xu wrote: > > Dear Flink devs, > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > > Connectors for the Apache Flink project[1] to the Apache Flink community. > > > > CDC Connectors for Apache Flink comprise a collection of source connectors > > designed specifically for Apache Flink. These connectors[2] enable the > > ingestion of changes from various databases using Change Data Capture > > (CDC), most of these CDC connectors are powered by Debezium[3]. They > > support both the DataStream API and the Table/SQL API, facilitating the > > reading of database snapshots and continuous reading of transaction logs > > with exactly-once processing, even in the event of failures. > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution[4]. > > > > > > I believe this initiative is a perfect match for both sides. For the Flink > > community, it presents an opportunity to enhance Flink's competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC aiming to streaming data > > intergration through Apache Flink, following the naming convention of > > Flink-ML; And this project is managed by a total of 8 maintainers, > > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > maintainers are also highly active contributors to the Flink community, > > donating this project to the Flink community implies that their permissions > > might be reduced. Therefore, we may need to bring up this topic for further > > discussion within the Flink PMC. Additionally, we need to discuss how to > > migrate existing users and documents. We have a user group of nearly 10,000 > > people and a multi-version documentation site need to migrate. We also need > > to plan for the migration of CI/CD processes and other specifics. > > > > > > While there are many intricate details that require implementation, we are > > committed to progressing and finalizing this donation process. > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I believe > > it's essential to commence discussions on future operations only after the > > community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > Best, > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > [1] https://github.com/ververica/flink-cdc-connectors > > [2] > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > [3] https://debezium.io > > [4] > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 for me, As a contributor in Flink CDC, I've witness so much excellent work that Flink CDC community have done for the latest years. Also Apache InLong benefits a lot on Flink CDC for its ability to support database changes. Much appreciation for the Flink CDC community and its maintainers. It's a great pleasure to support this contribution. Best, Zirui. On 2023/12/08 07:10:48 Yu Li wrote: > +1 > > Thanks Leonard for the proposal, and all Flink CDC contributors for the > existing great works. I believe these two projects are a perfect fit and > the donation will benifit both. > > Best Regards, > Yu > > > On Fri, Dec 8, 2023 at 2:49 PM bai_wentao1 wrote: > > > +1 for this exciting work > > > > > > Best regards, > > WT > > > > > > | | > > bai_wentao1 > > | > > | > > bai_went...@163.com > > | > > Replied Message ---- > > | From | ConradJam | > > | Date | 12/8/2023 14:26 | > > | To | | > > | Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to > > Apache Flink | > > +1 Best Idea . Thanks to all the contributors of Flink CDC for the great > > work. > > > > yuxia 于2023年12月7日周四 20:46写道： > > > > +1 for this. Thanks all the contributors of Flink CDC for the great work. > > > > Best regards, > > Yuxia > > > > - 原始邮件 - > > 发件人: "Martijn Visser" > > 收件人: "dev" > > 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36 > > 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink > > > > Hi Leonard, > > > > +1 for this. I think the CDC connectors are a great result and example > > of the Flink CDC community. Kudos to you all. > > > > Best regards, > > > > Martijn > > > > On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi > > wrote: > > > > Hi Leonard, > > > > Thank you for the excellent work you and the team working on the CDC > > connectors project have been doing so far. I am +1 of having them under > > Flink's umbrella. > > > > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot > > wrote: > > > > Big +1, thanks this will be a very useful addition to Flink. > > > > Best > > > > Etienne > > > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > > +1 for contributing CDC Connectors to Apache Flink. > > > > Best, > > Hang > > > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > > > Cool, +1 for contributing CDC Connectors to Apache Flink. > > > > Best, > > Yuxin > > > > > > Jing Ge 于2023年12月7日周四 15:43写道： > > > > Awesome! +1 > > > > Best regards, > > Jing > > > > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > > > wrote: > > > > thanks for working on this and driving it > > > > +1 > > > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > > wrote: > > > > This is incredibly exciting news, a big +1 for this. > > > > Thank you for the fantastic work on Flink CDC. We have created > > thousands > > of > > real-time integration jobs using Flink CDC connectors. > > > > > > Best, > > Feng > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > > gongzhongqi...@apache.org > > wrote: > > > > It's very exciting to hear the news. > > +1 for adding CDC Connectors to Apache Flink ! > > > > > > Best, > > Zhongqiang > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning > > to > > donate > > CDC Connectors for the Apache Flink project > > *[1]* to the Apache Flink community. > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > connectors designed specifically for Apache Flink. These > > connectors > > *[2]* > > enable the ingestion of changes from various databases using > > Change > > Data Capture (CDC), most of these CDC connectors are powered by > > Debezium > > *[3]* > > . They support both the DataStream API and the Table/SQL API, > > facilitating the reading of database snapshots and continuous > > reading > > of > > transaction logs with exactly-once processing, even in the > > event of > > failures. > > > > > > Additionally, in the latest version 3.0, we have introduced > > many > > long-awaited features. Starting from CDC version 3.0, we've > &g
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
good news. +1 Best, gongxin On 2023/12/07 03:24:59 Leonard Xu wrote: > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 Thanks Leonard for the proposal, and all Flink CDC contributors for the existing great works. I believe these two projects are a perfect fit and the donation will benifit both. Best Regards, Yu On Fri, Dec 8, 2023 at 2:49 PM bai_wentao1 wrote: > +1 for this exciting work > > > Best regards, > WT > > > | | > bai_wentao1 > | > | > bai_went...@163.com > | > Replied Message > | From | ConradJam | > | Date | 12/8/2023 14:26 | > | To | | > | Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to > Apache Flink | > +1 Best Idea . Thanks to all the contributors of Flink CDC for the great > work. > > yuxia 于2023年12月7日周四 20:46写道： > > +1 for this. Thanks all the contributors of Flink CDC for the great work. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Martijn Visser" > 收件人: "dev" > 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36 > 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink > > Hi Leonard, > > +1 for this. I think the CDC connectors are a great result and example > of the Flink CDC community. Kudos to you all. > > Best regards, > > Martijn > > On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi > wrote: > > Hi Leonard, > > Thank you for the excellent work you and the team working on the CDC > connectors project have been doing so far. I am +1 of having them under > Flink's umbrella. > > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot > wrote: > > Big +1, thanks this will be a very useful addition to Flink. > > Best > > Etienne > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > +1 for contributing CDC Connectors to Apache Flink. > > Best, > Hang > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > Cool, +1 for contributing CDC Connectors to Apache Flink. > > Best, > Yuxin > > > Jing Ge 于2023年12月7日周四 15:43写道： > > Awesome! +1 > > Best regards, > Jing > > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > wrote: > > thanks for working on this and driving it > > +1 > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > wrote: > > This is incredibly exciting news, a big +1 for this. > > Thank you for the fantastic work on Flink CDC. We have created > thousands > of > real-time integration jobs using Flink CDC connectors. > > > Best, > Feng > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > gongzhongqi...@apache.org > wrote: > > It's very exciting to hear the news. > +1 for adding CDC Connectors to Apache Flink ! > > > Best, > Zhongqiang > > Leonard Xu 于2023年12月7日周四 11:25写道： > > Dear Flink devs, > > > As you may have heard, we at Alibaba (Ververica) are planning > to > donate > CDC Connectors for the Apache Flink project > *[1]* to the Apache Flink community. > > > > CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These > connectors > *[2]* > enable the ingestion of changes from various databases using > Change > Data Capture (CDC), most of these CDC connectors are powered by > Debezium > *[3]* > . They support both the DataStream API and the Table/SQL API, > facilitating the reading of database snapshots and continuous > reading > of > transaction logs with exactly-once processing, even in the > event of > failures. > > > Additionally, in the latest version 3.0, we have introduced > many > long-awaited features. Starting from CDC version 3.0, we've > built a > Streaming ELT Framework available for streaming data > integration. > This > framework allows users to write their data synchronization logic > in a > simple YAML file, which will automatically be translated into a > Flink > DataStreaming job. It emphasizes optimizing the task submission > process > and > offers advanced functionalities such as whole database > synchronization, > merging sharded tables, and schema evolution > *[4]*. > > > > > I believe this initiative is a perfect match for both sides. > For > the > Flink community, it presents an opportunity to enhance Flink's > competitive > advantage in streaming data integration, promoting the healthy > growth > and > prosperity of the Apache Flink ecosystem. For the CDC Connectors > project, > becoming a sub-project of Apache Flink means being part of a > neutral > open-source community, which can attract a more diverse pool of > contributors. > > Please note that the aforementioned points represent only some > of > our > motivations and vision for this donation. Specific future > operations > need > to be fu
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 for this exciting work Best regards, WT | | bai_wentao1 | | bai_went...@163.com | Replied Message | From | ConradJam | | Date | 12/8/2023 14:26 | | To | | | Subject | Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink | +1 Best Idea . Thanks to all the contributors of Flink CDC for the great work. yuxia 于2023年12月7日周四 20:46写道： +1 for this. Thanks all the contributors of Flink CDC for the great work. Best regards, Yuxia - 原始邮件 - 发件人: "Martijn Visser" 收件人: "dev" 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink Hi Leonard, +1 for this. I think the CDC connectors are a great result and example of the Flink CDC community. Kudos to you all. Best regards, Martijn On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi wrote: Hi Leonard, Thank you for the excellent work you and the team working on the CDC connectors project have been doing so far. I am +1 of having them under Flink's umbrella. On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot wrote: Big +1, thanks this will be a very useful addition to Flink. Best Etienne Le 07/12/2023 à 09:26, Hang Ruan a écrit : +1 for contributing CDC Connectors to Apache Flink. Best, Hang Yuxin Tan 于2023年12月7日周四 16:05写道： Cool, +1 for contributing CDC Connectors to Apache Flink. Best, Yuxin Jing Ge 于2023年12月7日周四 15:43写道： Awesome! +1 Best regards, Jing On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin wrote: This is incredibly exciting news, a big +1 for this. Thank you for the fantastic work on Flink CDC. We have created thousands of real-time integration jobs using Flink CDC connectors. Best, Feng On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < gongzhongqi...@apache.org wrote: It's very exciting to hear the news. +1 for adding CDC Connectors to Apache Flink ! Best, Zhongqiang Leonard Xu 于2023年12月7日周四 11:25写道： Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project *[1]* to the Apache Flink community. CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors *[2]* enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium *[3]* . They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures. Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution *[4]*. I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors. Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process. Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 Best Idea . Thanks to all the contributors of Flink CDC for the great work. yuxia 于2023年12月7日周四 20:46写道： > +1 for this. Thanks all the contributors of Flink CDC for the great work. > > Best regards, > Yuxia > > - 原始邮件 - > 发件人: "Martijn Visser" > 收件人: "dev" > 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36 > 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink > > Hi Leonard, > > +1 for this. I think the CDC connectors are a great result and example > of the Flink CDC community. Kudos to you all. > > Best regards, > > Martijn > > On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi > wrote: > > > > Hi Leonard, > > > > Thank you for the excellent work you and the team working on the CDC > > connectors project have been doing so far. I am +1 of having them under > > Flink's umbrella. > > > > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot > > wrote: > > > > > Big +1, thanks this will be a very useful addition to Flink. > > > > > > Best > > > > > > Etienne > > > > > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > > > > +1 for contributing CDC Connectors to Apache Flink. > > > > > > > > Best, > > > > Hang > > > > > > > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > > > > > > >> Cool, +1 for contributing CDC Connectors to Apache Flink. > > > >> > > > >> Best, > > > >> Yuxin > > > >> > > > >> > > > >> Jing Ge 于2023年12月7日周四 15:43写道： > > > >> > > > >>> Awesome! +1 > > > >>> > > > >>> Best regards, > > > >>> Jing > > > >>> > > > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > > > > >>> wrote: > > > >>> > > > >>>> thanks for working on this and driving it > > > >>>> > > > >>>> +1 > > > >>>> > > > >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > > > wrote: > > > >>>> > > > >>>>> This is incredibly exciting news, a big +1 for this. > > > >>>>> > > > >>>>> Thank you for the fantastic work on Flink CDC. We have created > > > >>> thousands > > > >>>> of > > > >>>>> real-time integration jobs using Flink CDC connectors. > > > >>>>> > > > >>>>> > > > >>>>> Best, > > > >>>>> Feng > > > >>>>> > > > >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > > > >>> gongzhongqi...@apache.org > > > >>>>> wrote: > > > >>>>> > > > >>>>>> It's very exciting to hear the news. > > > >>>>>> +1 for adding CDC Connectors to Apache Flink ! > > > >>>>>> > > > >>>>>> > > > >>>>>> Best, > > > >>>>>> Zhongqiang > > > >>>>>> > > > >>>>>> Leonard Xu 于2023年12月7日周四 11:25写道： > > > >>>>>> > > > >>>>>>> Dear Flink devs, > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning > to > > > >>>> donate > > > >>>>>> CDC Connectors for the Apache Flink project > > > >>>>>>> *[1]* to the Apache Flink community. > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> > > > >>>>>>> CDC Connectors for Apache Flink comprise a collection of source > > > >>>>>> connectors designed specifically for Apache Flink. These > connectors > > > >>>>>>> *[2]* > > > >>>>>>> enable the ingestion of changes from various databases using > > > >>> Change > > > >>>>>> Data Capture (CDC), most of these CDC connectors are powered by > > > >>>> Debezium > > > >>>>>>> *[3]* > > > >>>>>>> . They support both the DataStream API
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 for this. Thanks all the contributors of Flink CDC for the great work. Best regards, Yuxia - 原始邮件 - 发件人: "Martijn Visser" 收件人: "dev" 发送时间: 星期四, 2023年 12 月 07日 下午 8:14:36 主题: Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink Hi Leonard, +1 for this. I think the CDC connectors are a great result and example of the Flink CDC community. Kudos to you all. Best regards, Martijn On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi wrote: > > Hi Leonard, > > Thank you for the excellent work you and the team working on the CDC > connectors project have been doing so far. I am +1 of having them under > Flink's umbrella. > > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot > wrote: > > > Big +1, thanks this will be a very useful addition to Flink. > > > > Best > > > > Etienne > > > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > > > +1 for contributing CDC Connectors to Apache Flink. > > > > > > Best, > > > Hang > > > > > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > > > > >> Cool, +1 for contributing CDC Connectors to Apache Flink. > > >> > > >> Best, > > >> Yuxin > > >> > > >> > > >> Jing Ge 于2023年12月7日周四 15:43写道： > > >> > > >>> Awesome! +1 > > >>> > > >>> Best regards, > > >>> Jing > > >>> > > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > > >>> wrote: > > >>> > > >>>> thanks for working on this and driving it > > >>>> > > >>>> +1 > > >>>> > > >>>> On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > > wrote: > > >>>> > > >>>>> This is incredibly exciting news, a big +1 for this. > > >>>>> > > >>>>> Thank you for the fantastic work on Flink CDC. We have created > > >>> thousands > > >>>> of > > >>>>> real-time integration jobs using Flink CDC connectors. > > >>>>> > > >>>>> > > >>>>> Best, > > >>>>> Feng > > >>>>> > > >>>>> On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > > >>> gongzhongqi...@apache.org > > >>>>> wrote: > > >>>>> > > >>>>>> It's very exciting to hear the news. > > >>>>>> +1 for adding CDC Connectors to Apache Flink ! > > >>>>>> > > >>>>>> > > >>>>>> Best, > > >>>>>> Zhongqiang > > >>>>>> > > >>>>>> Leonard Xu 于2023年12月7日周四 11:25写道： > > >>>>>> > > >>>>>>> Dear Flink devs, > > >>>>>>> > > >>>>>>> > > >>>>>>> As you may have heard, we at Alibaba (Ververica) are planning to > > >>>> donate > > >>>>>> CDC Connectors for the Apache Flink project > > >>>>>>> *[1]* to the Apache Flink community. > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> CDC Connectors for Apache Flink comprise a collection of source > > >>>>>> connectors designed specifically for Apache Flink. These connectors > > >>>>>>> *[2]* > > >>>>>>> enable the ingestion of changes from various databases using > > >>> Change > > >>>>>> Data Capture (CDC), most of these CDC connectors are powered by > > >>>> Debezium > > >>>>>>> *[3]* > > >>>>>>> . They support both the DataStream API and the Table/SQL API, > > >>>>>> facilitating the reading of database snapshots and continuous > > >> reading > > >>>> of > > >>>>>> transaction logs with exactly-once processing, even in the event of > > >>>>>> failures. > > >>>>>>> > > >>>>>>> > > >>>>>>> Additionally, in the latest version 3.0, we have introduced many > > >>>>>> long-awaited features. Starting from CDC version 3.0, we've built a > > >>>>>> Streaming ELT Framework available for
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Hi Leonard, +1 for this. I think the CDC connectors are a great result and example of the Flink CDC community. Kudos to you all. Best regards, Martijn On Thu, Dec 7, 2023 at 10:40 AM Márton Balassi wrote: > > Hi Leonard, > > Thank you for the excellent work you and the team working on the CDC > connectors project have been doing so far. I am +1 of having them under > Flink's umbrella. > > On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot > wrote: > > > Big +1, thanks this will be a very useful addition to Flink. > > > > Best > > > > Etienne > > > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > > > +1 for contributing CDC Connectors to Apache Flink. > > > > > > Best, > > > Hang > > > > > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > > > > >> Cool, +1 for contributing CDC Connectors to Apache Flink. > > >> > > >> Best, > > >> Yuxin > > >> > > >> > > >> Jing Ge 于2023年12月7日周四 15:43写道： > > >> > > >>> Awesome! +1 > > >>> > > >>> Best regards, > > >>> Jing > > >>> > > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > > >>> wrote: > > >>> > > thanks for working on this and driving it > > > > +1 > > > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > > wrote: > > > > > This is incredibly exciting news, a big +1 for this. > > > > > > Thank you for the fantastic work on Flink CDC. We have created > > >>> thousands > > of > > > real-time integration jobs using Flink CDC connectors. > > > > > > > > > Best, > > > Feng > > > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > > >>> gongzhongqi...@apache.org > > > wrote: > > > > > >> It's very exciting to hear the news. > > >> +1 for adding CDC Connectors to Apache Flink ! > > >> > > >> > > >> Best, > > >> Zhongqiang > > >> > > >> Leonard Xu 于2023年12月7日周四 11:25写道： > > >> > > >>> Dear Flink devs, > > >>> > > >>> > > >>> As you may have heard, we at Alibaba (Ververica) are planning to > > donate > > >> CDC Connectors for the Apache Flink project > > >>> *[1]* to the Apache Flink community. > > >>> > > >>> > > >>> > > >>> CDC Connectors for Apache Flink comprise a collection of source > > >> connectors designed specifically for Apache Flink. These connectors > > >>> *[2]* > > >>> enable the ingestion of changes from various databases using > > >>> Change > > >> Data Capture (CDC), most of these CDC connectors are powered by > > Debezium > > >>> *[3]* > > >>> . They support both the DataStream API and the Table/SQL API, > > >> facilitating the reading of database snapshots and continuous > > >> reading > > of > > >> transaction logs with exactly-once processing, even in the event of > > >> failures. > > >>> > > >>> > > >>> Additionally, in the latest version 3.0, we have introduced many > > >> long-awaited features. Starting from CDC version 3.0, we've built a > > >> Streaming ELT Framework available for streaming data integration. > > >>> This > > >> framework allows users to write their data synchronization logic > > >> in a > > >> simple YAML file, which will automatically be translated into a > > >> Flink > > >> DataStreaming job. It emphasizes optimizing the task submission > > >>> process > > > and > > >> offers advanced functionalities such as whole database > > >>> synchronization, > > >> merging sharded tables, and schema evolution > > >>> *[4]*. > > >>> > > >>> > > >>> > > >>> > > >>> I believe this initiative is a perfect match for both sides. For > > >>> the > > >> Flink community, it presents an opportunity to enhance Flink's > > > competitive > > >> advantage in streaming data integration, promoting the healthy > > >> growth > > and > > >> prosperity of the Apache Flink ecosystem. For the CDC Connectors > > project, > > >> becoming a sub-project of Apache Flink means being part of a > > >> neutral > > >> open-source community, which can attract a more diverse pool of > > >> contributors. > > >>> > > >>> Please note that the aforementioned points represent only some of > > >>> our > > >> motivations and vision for this donation. Specific future > > >> operations > > need > > >> to be further discussed in this thread. For example, the > > >> sub-project > > name > > >> after the donation; we hope to name it Flink-CDC > > >>> aiming to streaming data intergration through Apache Flink, > > >>> following the naming convention of Flink-ML; And this project is > > > managed > > >> by a total of 8 maintainers, including 3 Flink PMC members and 1 > > >>> Flink > > >> Committer. The remaining 4 maintainers are also highly active > > > contributors > > >> to the Flink community, donating this project to the Flink > > >> community > > >> implies that their permissions might be reduced. Therefore,
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Hi Leonard, Thank you for the excellent work you and the team working on the CDC connectors project have been doing so far. I am +1 of having them under Flink's umbrella. On Thu, Dec 7, 2023 at 10:26 AM Etienne Chauchot wrote: > Big +1, thanks this will be a very useful addition to Flink. > > Best > > Etienne > > Le 07/12/2023 à 09:26, Hang Ruan a écrit : > > +1 for contributing CDC Connectors to Apache Flink. > > > > Best, > > Hang > > > > Yuxin Tan 于2023年12月7日周四 16:05写道： > > > >> Cool, +1 for contributing CDC Connectors to Apache Flink. > >> > >> Best, > >> Yuxin > >> > >> > >> Jing Ge 于2023年12月7日周四 15:43写道： > >> > >>> Awesome! +1 > >>> > >>> Best regards, > >>> Jing > >>> > >>> On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > >>> wrote: > >>> > thanks for working on this and driving it > > +1 > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin > wrote: > > > This is incredibly exciting news, a big +1 for this. > > > > Thank you for the fantastic work on Flink CDC. We have created > >>> thousands > of > > real-time integration jobs using Flink CDC connectors. > > > > > > Best, > > Feng > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > >>> gongzhongqi...@apache.org > > wrote: > > > >> It's very exciting to hear the news. > >> +1 for adding CDC Connectors to Apache Flink ! > >> > >> > >> Best, > >> Zhongqiang > >> > >> Leonard Xu 于2023年12月7日周四 11:25写道： > >> > >>> Dear Flink devs, > >>> > >>> > >>> As you may have heard, we at Alibaba (Ververica) are planning to > donate > >> CDC Connectors for the Apache Flink project > >>> *[1]* to the Apache Flink community. > >>> > >>> > >>> > >>> CDC Connectors for Apache Flink comprise a collection of source > >> connectors designed specifically for Apache Flink. These connectors > >>> *[2]* > >>> enable the ingestion of changes from various databases using > >>> Change > >> Data Capture (CDC), most of these CDC connectors are powered by > Debezium > >>> *[3]* > >>> . They support both the DataStream API and the Table/SQL API, > >> facilitating the reading of database snapshots and continuous > >> reading > of > >> transaction logs with exactly-once processing, even in the event of > >> failures. > >>> > >>> > >>> Additionally, in the latest version 3.0, we have introduced many > >> long-awaited features. Starting from CDC version 3.0, we've built a > >> Streaming ELT Framework available for streaming data integration. > >>> This > >> framework allows users to write their data synchronization logic > >> in a > >> simple YAML file, which will automatically be translated into a > >> Flink > >> DataStreaming job. It emphasizes optimizing the task submission > >>> process > > and > >> offers advanced functionalities such as whole database > >>> synchronization, > >> merging sharded tables, and schema evolution > >>> *[4]*. > >>> > >>> > >>> > >>> > >>> I believe this initiative is a perfect match for both sides. For > >>> the > >> Flink community, it presents an opportunity to enhance Flink's > > competitive > >> advantage in streaming data integration, promoting the healthy > >> growth > and > >> prosperity of the Apache Flink ecosystem. For the CDC Connectors > project, > >> becoming a sub-project of Apache Flink means being part of a > >> neutral > >> open-source community, which can attract a more diverse pool of > >> contributors. > >>> > >>> Please note that the aforementioned points represent only some of > >>> our > >> motivations and vision for this donation. Specific future > >> operations > need > >> to be further discussed in this thread. For example, the > >> sub-project > name > >> after the donation; we hope to name it Flink-CDC > >>> aiming to streaming data intergration through Apache Flink, > >>> following the naming convention of Flink-ML; And this project is > > managed > >> by a total of 8 maintainers, including 3 Flink PMC members and 1 > >>> Flink > >> Committer. The remaining 4 maintainers are also highly active > > contributors > >> to the Flink community, donating this project to the Flink > >> community > >> implies that their permissions might be reduced. Therefore, we may > >>> need > > to > >> bring up this topic for further discussion within the Flink PMC. > >> Additionally, we need to discuss how to migrate existing users and > >> documents. We have a user group of nearly 10,000 people and a > > multi-version > >> documentation site need to migrate. We also need to plan for the > > migration > >> of CI/CD processes and other specifics. > >>> > >>> > >>> While there are many intricate details that require > >>
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Big +1, thanks this will be a very useful addition to Flink. Best Etienne Le 07/12/2023 à 09:26, Hang Ruan a écrit : +1 for contributing CDC Connectors to Apache Flink. Best, Hang Yuxin Tan 于2023年12月7日周四 16:05写道： Cool, +1 for contributing CDC Connectors to Apache Flink. Best, Yuxin Jing Ge 于2023年12月7日周四 15:43写道： Awesome! +1 Best regards, Jing On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin wrote: thanks for working on this and driving it +1 On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: This is incredibly exciting news, a big +1 for this. Thank you for the fantastic work on Flink CDC. We have created thousands of real-time integration jobs using Flink CDC connectors. Best, Feng On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < gongzhongqi...@apache.org wrote: It's very exciting to hear the news. +1 for adding CDC Connectors to Apache Flink ! Best, Zhongqiang Leonard Xu 于2023年12月7日周四 11:25写道： Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project *[1]* to the Apache Flink community. CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors *[2]* enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium *[3]* . They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures. Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution *[4]*. I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors. Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process. Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation. Really looking forward to hear what you think! Best, Leonard (on behalf of the Flink CDC Connectors project maintainers) [1]https://github.com/ververica/flink-cdc-connectors [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html [3]https://debezium.io [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html -- Best regards, Sergey
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 for contributing CDC Connectors to Apache Flink. Best, Hang Yuxin Tan 于2023年12月7日周四 16:05写道： > Cool, +1 for contributing CDC Connectors to Apache Flink. > > Best, > Yuxin > > > Jing Ge 于2023年12月7日周四 15:43写道： > > > Awesome! +1 > > > > Best regards, > > Jing > > > > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > > wrote: > > > > > thanks for working on this and driving it > > > > > > +1 > > > > > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > > > > > > > This is incredibly exciting news, a big +1 for this. > > > > > > > > Thank you for the fantastic work on Flink CDC. We have created > > thousands > > > of > > > > real-time integration jobs using Flink CDC connectors. > > > > > > > > > > > > Best, > > > > Feng > > > > > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > > gongzhongqi...@apache.org > > > > > > > > wrote: > > > > > > > > > It's very exciting to hear the news. > > > > > +1 for adding CDC Connectors to Apache Flink ! > > > > > > > > > > > > > > > Best, > > > > > Zhongqiang > > > > > > > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > > > > > > > > Dear Flink devs, > > > > > > > > > > > > > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to > > > donate > > > > > CDC Connectors for the Apache Flink project > > > > > > *[1]* to the Apache Flink community. > > > > > > > > > > > > > > > > > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > > > > connectors designed specifically for Apache Flink. These connectors > > > > > > *[2]* > > > > > > enable the ingestion of changes from various databases using > > Change > > > > > Data Capture (CDC), most of these CDC connectors are powered by > > > Debezium > > > > > > *[3]* > > > > > > . They support both the DataStream API and the Table/SQL API, > > > > > facilitating the reading of database snapshots and continuous > reading > > > of > > > > > transaction logs with exactly-once processing, even in the event of > > > > > failures. > > > > > > > > > > > > > > > > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > > > > long-awaited features. Starting from CDC version 3.0, we've built a > > > > > Streaming ELT Framework available for streaming data integration. > > This > > > > > framework allows users to write their data synchronization logic > in a > > > > > simple YAML file, which will automatically be translated into a > Flink > > > > > DataStreaming job. It emphasizes optimizing the task submission > > process > > > > and > > > > > offers advanced functionalities such as whole database > > synchronization, > > > > > merging sharded tables, and schema evolution > > > > > > *[4]*. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > I believe this initiative is a perfect match for both sides. For > > the > > > > > Flink community, it presents an opportunity to enhance Flink's > > > > competitive > > > > > advantage in streaming data integration, promoting the healthy > growth > > > and > > > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors > > > project, > > > > > becoming a sub-project of Apache Flink means being part of a > neutral > > > > > open-source community, which can attract a more diverse pool of > > > > > contributors. > > > > > > > > > > > > > > > > > > Please note that the aforementioned points represent only some of > > our > > > > > motivations and vision for this donation. Specific future > operations > > > need > > > > > to be further discussed in this thread. For example, the > sub-project > > > name > > > > > after the donation; we hope to name it Flink-CDC > > > > > > aiming to streaming data intergration through Apache Flink, > > > > > > following the naming convention of Flink-ML; And this project is > > > > managed > > > > > by a total of 8 maintainers, including 3 Flink PMC members and 1 > > Flink > > > > > Committer. The remaining 4 maintainers are also highly active > > > > contributors > > > > > to the Flink community, donating this project to the Flink > community > > > > > implies that their permissions might be reduced. Therefore, we may > > need > > > > to > > > > > bring up this topic for further discussion within the Flink PMC. > > > > > Additionally, we need to discuss how to migrate existing users and > > > > > documents. We have a user group of nearly 10,000 people and a > > > > multi-version > > > > > documentation site need to migrate. We also need to plan for the > > > > migration > > > > > of CI/CD processes and other specifics. > > > > > > > > > > > > > > > > > > > > > > > > While there are many intricate details that require > implementation, > > > we > > > > > are committed to progressing and finalizing this donation process. > > > > > > > > > > > > > > > > > > > > > > > > Despite being Flink’s most active ecological project (as > evaluated > > by > > > > > GitHub metrics), it also boasts a significant user base. However, I > > > > believe > > > > > it's essential to
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Cool, +1 for contributing CDC Connectors to Apache Flink. Best, Yuxin Jing Ge 于2023年12月7日周四 15:43写道： > Awesome! +1 > > Best regards, > Jing > > On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin > wrote: > > > thanks for working on this and driving it > > > > +1 > > > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > > > > > This is incredibly exciting news, a big +1 for this. > > > > > > Thank you for the fantastic work on Flink CDC. We have created > thousands > > of > > > real-time integration jobs using Flink CDC connectors. > > > > > > > > > Best, > > > Feng > > > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang < > gongzhongqi...@apache.org > > > > > > wrote: > > > > > > > It's very exciting to hear the news. > > > > +1 for adding CDC Connectors to Apache Flink ! > > > > > > > > > > > > Best, > > > > Zhongqiang > > > > > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > > > > > > Dear Flink devs, > > > > > > > > > > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to > > donate > > > > CDC Connectors for the Apache Flink project > > > > > *[1]* to the Apache Flink community. > > > > > > > > > > > > > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > > > connectors designed specifically for Apache Flink. These connectors > > > > > *[2]* > > > > > enable the ingestion of changes from various databases using > Change > > > > Data Capture (CDC), most of these CDC connectors are powered by > > Debezium > > > > > *[3]* > > > > > . They support both the DataStream API and the Table/SQL API, > > > > facilitating the reading of database snapshots and continuous reading > > of > > > > transaction logs with exactly-once processing, even in the event of > > > > failures. > > > > > > > > > > > > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > > > long-awaited features. Starting from CDC version 3.0, we've built a > > > > Streaming ELT Framework available for streaming data integration. > This > > > > framework allows users to write their data synchronization logic in a > > > > simple YAML file, which will automatically be translated into a Flink > > > > DataStreaming job. It emphasizes optimizing the task submission > process > > > and > > > > offers advanced functionalities such as whole database > synchronization, > > > > merging sharded tables, and schema evolution > > > > > *[4]*. > > > > > > > > > > > > > > > > > > > > > > > > > I believe this initiative is a perfect match for both sides. For > the > > > > Flink community, it presents an opportunity to enhance Flink's > > > competitive > > > > advantage in streaming data integration, promoting the healthy growth > > and > > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors > > project, > > > > becoming a sub-project of Apache Flink means being part of a neutral > > > > open-source community, which can attract a more diverse pool of > > > > contributors. > > > > > > > > > > > > > > > Please note that the aforementioned points represent only some of > our > > > > motivations and vision for this donation. Specific future operations > > need > > > > to be further discussed in this thread. For example, the sub-project > > name > > > > after the donation; we hope to name it Flink-CDC > > > > > aiming to streaming data intergration through Apache Flink, > > > > > following the naming convention of Flink-ML; And this project is > > > managed > > > > by a total of 8 maintainers, including 3 Flink PMC members and 1 > Flink > > > > Committer. The remaining 4 maintainers are also highly active > > > contributors > > > > to the Flink community, donating this project to the Flink community > > > > implies that their permissions might be reduced. Therefore, we may > need > > > to > > > > bring up this topic for further discussion within the Flink PMC. > > > > Additionally, we need to discuss how to migrate existing users and > > > > documents. We have a user group of nearly 10,000 people and a > > > multi-version > > > > documentation site need to migrate. We also need to plan for the > > > migration > > > > of CI/CD processes and other specifics. > > > > > > > > > > > > > > > > > > > > While there are many intricate details that require implementation, > > we > > > > are committed to progressing and finalizing this donation process. > > > > > > > > > > > > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated > by > > > > GitHub metrics), it also boasts a significant user base. However, I > > > believe > > > > it's essential to commence discussions on future operations only > after > > > the > > > > community reaches a consensus on whether they desire this donation. > > > > > > > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > > > > > > > > > > > > Best, > > > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > > > > > [1]
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Awesome! +1 Best regards, Jing On Thu, Dec 7, 2023 at 8:34 AM Sergey Nuyanzin wrote: > thanks for working on this and driving it > > +1 > > On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > > > This is incredibly exciting news, a big +1 for this. > > > > Thank you for the fantastic work on Flink CDC. We have created thousands > of > > real-time integration jobs using Flink CDC connectors. > > > > > > Best, > > Feng > > > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang > > > wrote: > > > > > It's very exciting to hear the news. > > > +1 for adding CDC Connectors to Apache Flink ! > > > > > > > > > Best, > > > Zhongqiang > > > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > > > > Dear Flink devs, > > > > > > > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to > donate > > > CDC Connectors for the Apache Flink project > > > > *[1]* to the Apache Flink community. > > > > > > > > > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > > connectors designed specifically for Apache Flink. These connectors > > > > *[2]* > > > > enable the ingestion of changes from various databases using Change > > > Data Capture (CDC), most of these CDC connectors are powered by > Debezium > > > > *[3]* > > > > . They support both the DataStream API and the Table/SQL API, > > > facilitating the reading of database snapshots and continuous reading > of > > > transaction logs with exactly-once processing, even in the event of > > > failures. > > > > > > > > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > > long-awaited features. Starting from CDC version 3.0, we've built a > > > Streaming ELT Framework available for streaming data integration. This > > > framework allows users to write their data synchronization logic in a > > > simple YAML file, which will automatically be translated into a Flink > > > DataStreaming job. It emphasizes optimizing the task submission process > > and > > > offers advanced functionalities such as whole database synchronization, > > > merging sharded tables, and schema evolution > > > > *[4]*. > > > > > > > > > > > > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > > > Flink community, it presents an opportunity to enhance Flink's > > competitive > > > advantage in streaming data integration, promoting the healthy growth > and > > > prosperity of the Apache Flink ecosystem. For the CDC Connectors > project, > > > becoming a sub-project of Apache Flink means being part of a neutral > > > open-source community, which can attract a more diverse pool of > > > contributors. > > > > > > > > > > > > Please note that the aforementioned points represent only some of our > > > motivations and vision for this donation. Specific future operations > need > > > to be further discussed in this thread. For example, the sub-project > name > > > after the donation; we hope to name it Flink-CDC > > > > aiming to streaming data intergration through Apache Flink, > > > > following the naming convention of Flink-ML; And this project is > > managed > > > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink > > > Committer. The remaining 4 maintainers are also highly active > > contributors > > > to the Flink community, donating this project to the Flink community > > > implies that their permissions might be reduced. Therefore, we may need > > to > > > bring up this topic for further discussion within the Flink PMC. > > > Additionally, we need to discuss how to migrate existing users and > > > documents. We have a user group of nearly 10,000 people and a > > multi-version > > > documentation site need to migrate. We also need to plan for the > > migration > > > of CI/CD processes and other specifics. > > > > > > > > > > > > > > > > While there are many intricate details that require implementation, > we > > > are committed to progressing and finalizing this donation process. > > > > > > > > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > > GitHub metrics), it also boasts a significant user base. However, I > > believe > > > it's essential to commence discussions on future operations only after > > the > > > community reaches a consensus on whether they desire this donation. > > > > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > > > > > > > > Best, > > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > > [2] > > > > > > > > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > > [3] https://debezium.io > > > > [4] > > > > > > > > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > > > > > > > > > > > -- > Best regards, > Sergey >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
thanks for working on this and driving it +1 On Thu, Dec 7, 2023 at 7:26 AM Feng Jin wrote: > This is incredibly exciting news, a big +1 for this. > > Thank you for the fantastic work on Flink CDC. We have created thousands of > real-time integration jobs using Flink CDC connectors. > > > Best, > Feng > > On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang > wrote: > > > It's very exciting to hear the news. > > +1 for adding CDC Connectors to Apache Flink ! > > > > > > Best, > > Zhongqiang > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > > Dear Flink devs, > > > > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > > CDC Connectors for the Apache Flink project > > > *[1]* to the Apache Flink community. > > > > > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > connectors designed specifically for Apache Flink. These connectors > > > *[2]* > > > enable the ingestion of changes from various databases using Change > > Data Capture (CDC), most of these CDC connectors are powered by Debezium > > > *[3]* > > > . They support both the DataStream API and the Table/SQL API, > > facilitating the reading of database snapshots and continuous reading of > > transaction logs with exactly-once processing, even in the event of > > failures. > > > > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process > and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution > > > *[4]*. > > > > > > > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > > Flink community, it presents an opportunity to enhance Flink's > competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > > > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC > > > aiming to streaming data intergration through Apache Flink, > > > following the naming convention of Flink-ML; And this project is > managed > > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink > > Committer. The remaining 4 maintainers are also highly active > contributors > > to the Flink community, donating this project to the Flink community > > implies that their permissions might be reduced. Therefore, we may need > to > > bring up this topic for further discussion within the Flink PMC. > > Additionally, we need to discuss how to migrate existing users and > > documents. We have a user group of nearly 10,000 people and a > multi-version > > documentation site need to migrate. We also need to plan for the > migration > > of CI/CD processes and other specifics. > > > > > > > > > > > > While there are many intricate details that require implementation, we > > are committed to progressing and finalizing this donation process. > > > > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I > believe > > it's essential to commence discussions on future operations only after > the > > community reaches a consensus on whether they desire this donation. > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > > > > Best, > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > [2] > > > > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > [3] https://debezium.io > > > [4] > > > > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > > > > > -- Best regards, Sergey
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
This is incredibly exciting news, a big +1 for this. Thank you for the fantastic work on Flink CDC. We have created thousands of real-time integration jobs using Flink CDC connectors. Best, Feng On Thu, Dec 7, 2023 at 1:45 PM gongzhongqiang wrote: > It's very exciting to hear the news. > +1 for adding CDC Connectors to Apache Flink ! > > > Best, > Zhongqiang > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > CDC Connectors for the Apache Flink project > > *[1]* to the Apache Flink community. > > > > > > > > CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These connectors > > *[2]* > > enable the ingestion of changes from various databases using Change > Data Capture (CDC), most of these CDC connectors are powered by Debezium > > *[3]* > > . They support both the DataStream API and the Table/SQL API, > facilitating the reading of database snapshots and continuous reading of > transaction logs with exactly-once processing, even in the event of > failures. > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > long-awaited features. Starting from CDC version 3.0, we've built a > Streaming ELT Framework available for streaming data integration. This > framework allows users to write their data synchronization logic in a > simple YAML file, which will automatically be translated into a Flink > DataStreaming job. It emphasizes optimizing the task submission process and > offers advanced functionalities such as whole database synchronization, > merging sharded tables, and schema evolution > > *[4]*. > > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > Flink community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of > contributors. > > > > > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need > to be further discussed in this thread. For example, the sub-project name > after the donation; we hope to name it Flink-CDC > > aiming to streaming data intergration through Apache Flink, > > following the naming convention of Flink-ML; And this project is managed > by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink > Committer. The remaining 4 maintainers are also highly active contributors > to the Flink community, donating this project to the Flink community > implies that their permissions might be reduced. Therefore, we may need to > bring up this topic for further discussion within the Flink PMC. > Additionally, we need to discuss how to migrate existing users and > documents. We have a user group of nearly 10,000 people and a multi-version > documentation site need to migrate. We also need to plan for the migration > of CI/CD processes and other specifics. > > > > > > > > While there are many intricate details that require implementation, we > are committed to progressing and finalizing this donation process. > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > GitHub metrics), it also boasts a significant user base. However, I believe > it's essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > > > Best, > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > [1] https://github.com/ververica/flink-cdc-connectors > > [2] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > [3] https://debezium.io > > [4] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
It's very exciting to hear the news. +1 for adding CDC Connectors to Apache Flink ! Best, Zhongqiang Leonard Xu 于2023年12月7日周四 11:25写道： > Dear Flink devs, > > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project > *[1]* to the Apache Flink community. > > > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors > *[2]* > enable the ingestion of changes from various databases using Change Data > Capture (CDC), most of these CDC connectors are powered by Debezium > *[3]* > . They support both the DataStream API and the Table/SQL API, facilitating > the reading of database snapshots and continuous reading of transaction logs > with exactly-once processing, even in the event of failures. > > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution > *[4]*. > > > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC > aiming to streaming data intergration through Apache Flink, > following the naming convention of Flink-ML; And this project is managed by a > total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. > The remaining 4 maintainers are also highly active contributors to the Flink > community, donating this project to the Flink community implies that their > permissions might be reduced. Therefore, we may need to bring up this topic > for further discussion within the Flink PMC. Additionally, we need to discuss > how to migrate existing users and documents. We have a user group of nearly > 10,000 people and a multi-version documentation site need to migrate. We also > need to plan for the migration of CI/CD processes and other specifics. > > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Very excited to hear the news, +1 for this proposal. Best, Jiabao > 2023年12月7日 13:08，Rui Fan <1996fan...@gmail.com> 写道： > > It's cool! > > Thanks for the great work of CDC, > +1 for this donation. > > Best, > Rui > > On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang > wrote: > >> So cool, Big +1 for this exciting work. >> >> Best >> Hongshun >> >> On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren wrote: >> >>> Thanks for kicking this off, Leonard! >>> >>> As one of the contributors of the CDC project, I'm truly honored to be >> part >>> of the community and so excited to hear the news. CDC project was born >> from >>> and developed together with Apache Flink, and we are so proud to be >>> accepted by more and more users around the world. >>> >>> To put my Flink hat on, I believe having the CDC project as a part of >>> Apache Flink will broadly expand our ecosystem and the usage scenarios. >>> Both two projects will benefit from closer cooperation, so +1 from my >> side. >>> >>> Best, >>> Qingsheng >>> >>> On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb >> wrote: >>> That's really cool :) +1 for the great addition Bests, Samrat On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li >>> wrote: > Wow, Cool, Nice > > CDC is playing an increasingly important role. > > +1 > > Best, > Jingsong > > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: >> >> Dear Flink devs, >> >> As you may have heard, we at Alibaba (Ververica) are planning to >>> donate > CDC Connectors for the Apache Flink project[1] to the Apache Flink > community. >> >> CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These connectors[2] > enable the ingestion of changes from various databases using Change >> Data > Capture (CDC), most of these CDC connectors are powered by >> Debezium[3]. > They support both the DataStream API and the Table/SQL API, >> facilitating > the reading of database snapshots and continuous reading of >> transaction > logs with exactly-once processing, even in the event of failures. >> >> >> Additionally, in the latest version 3.0, we have introduced many > long-awaited features. Starting from CDC version 3.0, we've built a > Streaming ELT Framework available for streaming data integration. This > framework allows users to write their data synchronization logic in a > simple YAML file, which will automatically be translated into a Flink > DataStreaming job. It emphasizes optimizing the task submission >> process >>> and > offers advanced functionalities such as whole database >> synchronization, > merging sharded tables, and schema evolution[4]. >> >> >> I believe this initiative is a perfect match for both sides. For the > Flink community, it presents an opportunity to enhance Flink's >>> competitive > advantage in streaming data integration, promoting the healthy growth >>> and > prosperity of the Apache Flink ecosystem. For the CDC Connectors >>> project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of > contributors. >> >> Please note that the aforementioned points represent only some of >> our > motivations and vision for this donation. Specific future operations >>> need > to be further discussed in this thread. For example, the sub-project >>> name > after the donation; we hope to name it Flink-CDC aiming to streaming >>> data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > maintainers are also highly active contributors to the Flink >> community, > donating this project to the Flink community implies that their >>> permissions > might be reduced. Therefore, we may need to bring up this topic for >>> further > discussion within the Flink PMC. Additionally, we need to discuss how >> to > migrate existing users and documents. We have a user group of nearly >>> 10,000 > people and a multi-version documentation site need to migrate. We also >>> need > to plan for the migration of CI/CD processes and other specifics. >> >> >> While there are many intricate details that require implementation, >> we > are committed to progressing and finalizing this donation process. >> >> >> Despite being Flink’s most active ecological project (as evaluated >> by > GitHub metrics), it also boasts a significant user base. However, I >>> believe > it's essential to commence discussions on future operations only after >>> the > community reaches a consensus on whether they desire this donation.
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
It's cool! Thanks for the great work of CDC, +1 for this donation. Best, Rui On Thu, Dec 7, 2023 at 12:28 PM Hongshun Wang wrote: > So cool, Big +1 for this exciting work. > > Best > Hongshun > > On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren wrote: > > > Thanks for kicking this off, Leonard! > > > > As one of the contributors of the CDC project, I'm truly honored to be > part > > of the community and so excited to hear the news. CDC project was born > from > > and developed together with Apache Flink, and we are so proud to be > > accepted by more and more users around the world. > > > > To put my Flink hat on, I believe having the CDC project as a part of > > Apache Flink will broadly expand our ecosystem and the usage scenarios. > > Both two projects will benefit from closer cooperation, so +1 from my > side. > > > > Best, > > Qingsheng > > > > On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb > wrote: > > > > > That's really cool :) > > > +1 for the great addition > > > > > > Bests, > > > Samrat > > > > > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li > > wrote: > > > > > >> Wow, Cool, Nice > > >> > > >> CDC is playing an increasingly important role. > > >> > > >> +1 > > >> > > >> Best, > > >> Jingsong > > >> > > >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > >> > > > >> > Dear Flink devs, > > >> > > > >> > As you may have heard, we at Alibaba (Ververica) are planning to > > donate > > >> CDC Connectors for the Apache Flink project[1] to the Apache Flink > > >> community. > > >> > > > >> > CDC Connectors for Apache Flink comprise a collection of source > > >> connectors designed specifically for Apache Flink. These connectors[2] > > >> enable the ingestion of changes from various databases using Change > Data > > >> Capture (CDC), most of these CDC connectors are powered by > Debezium[3]. > > >> They support both the DataStream API and the Table/SQL API, > facilitating > > >> the reading of database snapshots and continuous reading of > transaction > > >> logs with exactly-once processing, even in the event of failures. > > >> > > > >> > > > >> > Additionally, in the latest version 3.0, we have introduced many > > >> long-awaited features. Starting from CDC version 3.0, we've built a > > >> Streaming ELT Framework available for streaming data integration. This > > >> framework allows users to write their data synchronization logic in a > > >> simple YAML file, which will automatically be translated into a Flink > > >> DataStreaming job. It emphasizes optimizing the task submission > process > > and > > >> offers advanced functionalities such as whole database > synchronization, > > >> merging sharded tables, and schema evolution[4]. > > >> > > > >> > > > >> > I believe this initiative is a perfect match for both sides. For the > > >> Flink community, it presents an opportunity to enhance Flink's > > competitive > > >> advantage in streaming data integration, promoting the healthy growth > > and > > >> prosperity of the Apache Flink ecosystem. For the CDC Connectors > > project, > > >> becoming a sub-project of Apache Flink means being part of a neutral > > >> open-source community, which can attract a more diverse pool of > > >> contributors. > > >> > > > >> > Please note that the aforementioned points represent only some of > our > > >> motivations and vision for this donation. Specific future operations > > need > > >> to be further discussed in this thread. For example, the sub-project > > name > > >> after the donation; we hope to name it Flink-CDC aiming to streaming > > data > > >> intergration through Apache Flink, following the naming convention of > > >> Flink-ML; And this project is managed by a total of 8 maintainers, > > >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > >> maintainers are also highly active contributors to the Flink > community, > > >> donating this project to the Flink community implies that their > > permissions > > >> might be reduced. Therefore, we may need to bring up this topic for > > further > > >> discussion within the Flink PMC. Additionally, we need to discuss how > to > > >> migrate existing users and documents. We have a user group of nearly > > 10,000 > > >> people and a multi-version documentation site need to migrate. We also > > need > > >> to plan for the migration of CI/CD processes and other specifics. > > >> > > > >> > > > >> > While there are many intricate details that require implementation, > we > > >> are committed to progressing and finalizing this donation process. > > >> > > > >> > > > >> > Despite being Flink’s most active ecological project (as evaluated > by > > >> GitHub metrics), it also boasts a significant user base. However, I > > believe > > >> it's essential to commence discussions on future operations only after > > the > > >> community reaches a consensus on whether they desire this donation. > > >> > > > >> > > > >> > Really looking forward to hear what you think! > > >> > > > >> > > > >> > Best,
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
So cool, Big +1 for this exciting work. Best Hongshun On Thu, Dec 7, 2023 at 12:20 PM Qingsheng Ren wrote: > Thanks for kicking this off, Leonard! > > As one of the contributors of the CDC project, I'm truly honored to be part > of the community and so excited to hear the news. CDC project was born from > and developed together with Apache Flink, and we are so proud to be > accepted by more and more users around the world. > > To put my Flink hat on, I believe having the CDC project as a part of > Apache Flink will broadly expand our ecosystem and the usage scenarios. > Both two projects will benefit from closer cooperation, so +1 from my side. > > Best, > Qingsheng > > On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb wrote: > > > That's really cool :) > > +1 for the great addition > > > > Bests, > > Samrat > > > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li > wrote: > > > >> Wow, Cool, Nice > >> > >> CDC is playing an increasingly important role. > >> > >> +1 > >> > >> Best, > >> Jingsong > >> > >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > >> > > >> > Dear Flink devs, > >> > > >> > As you may have heard, we at Alibaba (Ververica) are planning to > donate > >> CDC Connectors for the Apache Flink project[1] to the Apache Flink > >> community. > >> > > >> > CDC Connectors for Apache Flink comprise a collection of source > >> connectors designed specifically for Apache Flink. These connectors[2] > >> enable the ingestion of changes from various databases using Change Data > >> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > >> They support both the DataStream API and the Table/SQL API, facilitating > >> the reading of database snapshots and continuous reading of transaction > >> logs with exactly-once processing, even in the event of failures. > >> > > >> > > >> > Additionally, in the latest version 3.0, we have introduced many > >> long-awaited features. Starting from CDC version 3.0, we've built a > >> Streaming ELT Framework available for streaming data integration. This > >> framework allows users to write their data synchronization logic in a > >> simple YAML file, which will automatically be translated into a Flink > >> DataStreaming job. It emphasizes optimizing the task submission process > and > >> offers advanced functionalities such as whole database synchronization, > >> merging sharded tables, and schema evolution[4]. > >> > > >> > > >> > I believe this initiative is a perfect match for both sides. For the > >> Flink community, it presents an opportunity to enhance Flink's > competitive > >> advantage in streaming data integration, promoting the healthy growth > and > >> prosperity of the Apache Flink ecosystem. For the CDC Connectors > project, > >> becoming a sub-project of Apache Flink means being part of a neutral > >> open-source community, which can attract a more diverse pool of > >> contributors. > >> > > >> > Please note that the aforementioned points represent only some of our > >> motivations and vision for this donation. Specific future operations > need > >> to be further discussed in this thread. For example, the sub-project > name > >> after the donation; we hope to name it Flink-CDC aiming to streaming > data > >> intergration through Apache Flink, following the naming convention of > >> Flink-ML; And this project is managed by a total of 8 maintainers, > >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > >> maintainers are also highly active contributors to the Flink community, > >> donating this project to the Flink community implies that their > permissions > >> might be reduced. Therefore, we may need to bring up this topic for > further > >> discussion within the Flink PMC. Additionally, we need to discuss how to > >> migrate existing users and documents. We have a user group of nearly > 10,000 > >> people and a multi-version documentation site need to migrate. We also > need > >> to plan for the migration of CI/CD processes and other specifics. > >> > > >> > > >> > While there are many intricate details that require implementation, we > >> are committed to progressing and finalizing this donation process. > >> > > >> > > >> > Despite being Flink’s most active ecological project (as evaluated by > >> GitHub metrics), it also boasts a significant user base. However, I > believe > >> it's essential to commence discussions on future operations only after > the > >> community reaches a consensus on whether they desire this donation. > >> > > >> > > >> > Really looking forward to hear what you think! > >> > > >> > > >> > Best, > >> > Leonard (on behalf of the Flink CDC Connectors project maintainers) > >> > > >> > [1] https://github.com/ververica/flink-cdc-connectors > >> > [2] > >> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > >> > [3] https://debezium.io > >> > [4] > >> > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > >> > >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Thanks for kicking this off, Leonard! As one of the contributors of the CDC project, I'm truly honored to be part of the community and so excited to hear the news. CDC project was born from and developed together with Apache Flink, and we are so proud to be accepted by more and more users around the world. To put my Flink hat on, I believe having the CDC project as a part of Apache Flink will broadly expand our ecosystem and the usage scenarios. Both two projects will benefit from closer cooperation, so +1 from my side. Best, Qingsheng On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > >> Wow, Cool, Nice >> >> CDC is playing an increasingly important role. >> >> +1 >> >> Best, >> Jingsong >> >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: >> > >> > Dear Flink devs, >> > >> > As you may have heard, we at Alibaba (Ververica) are planning to donate >> CDC Connectors for the Apache Flink project[1] to the Apache Flink >> community. >> > >> > CDC Connectors for Apache Flink comprise a collection of source >> connectors designed specifically for Apache Flink. These connectors[2] >> enable the ingestion of changes from various databases using Change Data >> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. >> They support both the DataStream API and the Table/SQL API, facilitating >> the reading of database snapshots and continuous reading of transaction >> logs with exactly-once processing, even in the event of failures. >> > >> > >> > Additionally, in the latest version 3.0, we have introduced many >> long-awaited features. Starting from CDC version 3.0, we've built a >> Streaming ELT Framework available for streaming data integration. This >> framework allows users to write their data synchronization logic in a >> simple YAML file, which will automatically be translated into a Flink >> DataStreaming job. It emphasizes optimizing the task submission process and >> offers advanced functionalities such as whole database synchronization, >> merging sharded tables, and schema evolution[4]. >> > >> > >> > I believe this initiative is a perfect match for both sides. For the >> Flink community, it presents an opportunity to enhance Flink's competitive >> advantage in streaming data integration, promoting the healthy growth and >> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, >> becoming a sub-project of Apache Flink means being part of a neutral >> open-source community, which can attract a more diverse pool of >> contributors. >> > >> > Please note that the aforementioned points represent only some of our >> motivations and vision for this donation. Specific future operations need >> to be further discussed in this thread. For example, the sub-project name >> after the donation; we hope to name it Flink-CDC aiming to streaming data >> intergration through Apache Flink, following the naming convention of >> Flink-ML; And this project is managed by a total of 8 maintainers, >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 >> maintainers are also highly active contributors to the Flink community, >> donating this project to the Flink community implies that their permissions >> might be reduced. Therefore, we may need to bring up this topic for further >> discussion within the Flink PMC. Additionally, we need to discuss how to >> migrate existing users and documents. We have a user group of nearly 10,000 >> people and a multi-version documentation site need to migrate. We also need >> to plan for the migration of CI/CD processes and other specifics. >> > >> > >> > While there are many intricate details that require implementation, we >> are committed to progressing and finalizing this donation process. >> > >> > >> > Despite being Flink’s most active ecological project (as evaluated by >> GitHub metrics), it also boasts a significant user base. However, I believe >> it's essential to commence discussions on future operations only after the >> community reaches a consensus on whether they desire this donation. >> > >> > >> > Really looking forward to hear what you think! >> > >> > >> > Best, >> > Leonard (on behalf of the Flink CDC Connectors project maintainers) >> > >> > [1] https://github.com/ververica/flink-cdc-connectors >> > [2] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html >> > [3] https://debezium.io >> > [4] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >> >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
+1 for adding this to Apache Flink! I think this can further extend the ability of Apache Flink and a lot of users would be interested to try this out. Best, Jark On Thu, 7 Dec 2023 at 12:06, Samrat Deb wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > >> Wow, Cool, Nice >> >> CDC is playing an increasingly important role. >> >> +1 >> >> Best, >> Jingsong >> >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: >> > >> > Dear Flink devs, >> > >> > As you may have heard, we at Alibaba (Ververica) are planning to donate >> CDC Connectors for the Apache Flink project[1] to the Apache Flink >> community. >> > >> > CDC Connectors for Apache Flink comprise a collection of source >> connectors designed specifically for Apache Flink. These connectors[2] >> enable the ingestion of changes from various databases using Change Data >> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. >> They support both the DataStream API and the Table/SQL API, facilitating >> the reading of database snapshots and continuous reading of transaction >> logs with exactly-once processing, even in the event of failures. >> > >> > >> > Additionally, in the latest version 3.0, we have introduced many >> long-awaited features. Starting from CDC version 3.0, we've built a >> Streaming ELT Framework available for streaming data integration. This >> framework allows users to write their data synchronization logic in a >> simple YAML file, which will automatically be translated into a Flink >> DataStreaming job. It emphasizes optimizing the task submission process and >> offers advanced functionalities such as whole database synchronization, >> merging sharded tables, and schema evolution[4]. >> > >> > >> > I believe this initiative is a perfect match for both sides. For the >> Flink community, it presents an opportunity to enhance Flink's competitive >> advantage in streaming data integration, promoting the healthy growth and >> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, >> becoming a sub-project of Apache Flink means being part of a neutral >> open-source community, which can attract a more diverse pool of >> contributors. >> > >> > Please note that the aforementioned points represent only some of our >> motivations and vision for this donation. Specific future operations need >> to be further discussed in this thread. For example, the sub-project name >> after the donation; we hope to name it Flink-CDC aiming to streaming data >> intergration through Apache Flink, following the naming convention of >> Flink-ML; And this project is managed by a total of 8 maintainers, >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 >> maintainers are also highly active contributors to the Flink community, >> donating this project to the Flink community implies that their permissions >> might be reduced. Therefore, we may need to bring up this topic for further >> discussion within the Flink PMC. Additionally, we need to discuss how to >> migrate existing users and documents. We have a user group of nearly 10,000 >> people and a multi-version documentation site need to migrate. We also need >> to plan for the migration of CI/CD processes and other specifics. >> > >> > >> > While there are many intricate details that require implementation, we >> are committed to progressing and finalizing this donation process. >> > >> > >> > Despite being Flink’s most active ecological project (as evaluated by >> GitHub metrics), it also boasts a significant user base. However, I believe >> it's essential to commence discussions on future operations only after the >> community reaches a consensus on whether they desire this donation. >> > >> > >> > Really looking forward to hear what you think! >> > >> > >> > Best, >> > Leonard (on behalf of the Flink CDC Connectors project maintainers) >> > >> > [1] https://github.com/ververica/flink-cdc-connectors >> > [2] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html >> > [3] https://debezium.io >> > [4] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >> >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
A big +1 for this proposal. Thanks Leonard and the Flink CDC community. While there are many further details to be discussed, I believe this proposal is aligned with the long-term interests for both communities. Best, Xintong On Thu, Dec 7, 2023 at 12:06 PM Samrat Deb wrote: > That's really cool :) > +1 for the great addition > > Bests, > Samrat > > On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > > > Wow, Cool, Nice > > > > CDC is playing an increasingly important role. > > > > +1 > > > > Best, > > Jingsong > > > > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > > > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > > CDC Connectors for the Apache Flink project[1] to the Apache Flink > > community. > > > > > > CDC Connectors for Apache Flink comprise a collection of source > > connectors designed specifically for Apache Flink. These connectors[2] > > enable the ingestion of changes from various databases using Change Data > > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > > They support both the DataStream API and the Table/SQL API, facilitating > > the reading of database snapshots and continuous reading of transaction > > logs with exactly-once processing, even in the event of failures. > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process > and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution[4]. > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > > Flink community, it presents an opportunity to enhance Flink's > competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC aiming to streaming data > > intergration through Apache Flink, following the naming convention of > > Flink-ML; And this project is managed by a total of 8 maintainers, > > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > maintainers are also highly active contributors to the Flink community, > > donating this project to the Flink community implies that their > permissions > > might be reduced. Therefore, we may need to bring up this topic for > further > > discussion within the Flink PMC. Additionally, we need to discuss how to > > migrate existing users and documents. We have a user group of nearly > 10,000 > > people and a multi-version documentation site need to migrate. We also > need > > to plan for the migration of CI/CD processes and other specifics. > > > > > > > > > While there are many intricate details that require implementation, we > > are committed to progressing and finalizing this donation process. > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I > believe > > it's essential to commence discussions on future operations only after > the > > community reaches a consensus on whether they desire this donation. > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > Best, > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > [2] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > [3] https://debezium.io > > > [4] > > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > >
Re:Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Big +1 for this exciting work. -- Best！ Xuyang 在 2023-12-07 12:06:07，"Samrat Deb" 写道： >That's really cool :) >+1 for the great addition > >Bests, >Samrat > >On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > >> Wow, Cool, Nice >> >> CDC is playing an increasingly important role. >> >> +1 >> >> Best, >> Jingsong >> >> On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: >> > >> > Dear Flink devs, >> > >> > As you may have heard, we at Alibaba (Ververica) are planning to donate >> CDC Connectors for the Apache Flink project[1] to the Apache Flink >> community. >> > >> > CDC Connectors for Apache Flink comprise a collection of source >> connectors designed specifically for Apache Flink. These connectors[2] >> enable the ingestion of changes from various databases using Change Data >> Capture (CDC), most of these CDC connectors are powered by Debezium[3]. >> They support both the DataStream API and the Table/SQL API, facilitating >> the reading of database snapshots and continuous reading of transaction >> logs with exactly-once processing, even in the event of failures. >> > >> > >> > Additionally, in the latest version 3.0, we have introduced many >> long-awaited features. Starting from CDC version 3.0, we've built a >> Streaming ELT Framework available for streaming data integration. This >> framework allows users to write their data synchronization logic in a >> simple YAML file, which will automatically be translated into a Flink >> DataStreaming job. It emphasizes optimizing the task submission process and >> offers advanced functionalities such as whole database synchronization, >> merging sharded tables, and schema evolution[4]. >> > >> > >> > I believe this initiative is a perfect match for both sides. For the >> Flink community, it presents an opportunity to enhance Flink's competitive >> advantage in streaming data integration, promoting the healthy growth and >> prosperity of the Apache Flink ecosystem. For the CDC Connectors project, >> becoming a sub-project of Apache Flink means being part of a neutral >> open-source community, which can attract a more diverse pool of >> contributors. >> > >> > Please note that the aforementioned points represent only some of our >> motivations and vision for this donation. Specific future operations need >> to be further discussed in this thread. For example, the sub-project name >> after the donation; we hope to name it Flink-CDC aiming to streaming data >> intergration through Apache Flink, following the naming convention of >> Flink-ML; And this project is managed by a total of 8 maintainers, >> including 3 Flink PMC members and 1 Flink Committer. The remaining 4 >> maintainers are also highly active contributors to the Flink community, >> donating this project to the Flink community implies that their permissions >> might be reduced. Therefore, we may need to bring up this topic for further >> discussion within the Flink PMC. Additionally, we need to discuss how to >> migrate existing users and documents. We have a user group of nearly 10,000 >> people and a multi-version documentation site need to migrate. We also need >> to plan for the migration of CI/CD processes and other specifics. >> > >> > >> > While there are many intricate details that require implementation, we >> are committed to progressing and finalizing this donation process. >> > >> > >> > Despite being Flink’s most active ecological project (as evaluated by >> GitHub metrics), it also boasts a significant user base. However, I believe >> it's essential to commence discussions on future operations only after the >> community reaches a consensus on whether they desire this donation. >> > >> > >> > Really looking forward to hear what you think! >> > >> > >> > Best, >> > Leonard (on behalf of the Flink CDC Connectors project maintainers) >> > >> > [1] https://github.com/ververica/flink-cdc-connectors >> > [2] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html >> > [3] https://debezium.io >> > [4] >> https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >>
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
That's really cool :) +1 for the great addition Bests, Samrat On Thu, 7 Dec 2023 at 9:20 AM, Jingsong Li wrote: > Wow, Cool, Nice > > CDC is playing an increasingly important role. > > +1 > > Best, > Jingsong > > On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > > > Dear Flink devs, > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate > CDC Connectors for the Apache Flink project[1] to the Apache Flink > community. > > > > CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These connectors[2] > enable the ingestion of changes from various databases using Change Data > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > They support both the DataStream API and the Table/SQL API, facilitating > the reading of database snapshots and continuous reading of transaction > logs with exactly-once processing, even in the event of failures. > > > > > > Additionally, in the latest version 3.0, we have introduced many > long-awaited features. Starting from CDC version 3.0, we've built a > Streaming ELT Framework available for streaming data integration. This > framework allows users to write their data synchronization logic in a > simple YAML file, which will automatically be translated into a Flink > DataStreaming job. It emphasizes optimizing the task submission process and > offers advanced functionalities such as whole database synchronization, > merging sharded tables, and schema evolution[4]. > > > > > > I believe this initiative is a perfect match for both sides. For the > Flink community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of > contributors. > > > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need > to be further discussed in this thread. For example, the sub-project name > after the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > maintainers are also highly active contributors to the Flink community, > donating this project to the Flink community implies that their permissions > might be reduced. Therefore, we may need to bring up this topic for further > discussion within the Flink PMC. Additionally, we need to discuss how to > migrate existing users and documents. We have a user group of nearly 10,000 > people and a multi-version documentation site need to migrate. We also need > to plan for the migration of CI/CD processes and other specifics. > > > > > > While there are many intricate details that require implementation, we > are committed to progressing and finalizing this donation process. > > > > > > Despite being Flink’s most active ecological project (as evaluated by > GitHub metrics), it also boasts a significant user base. However, I believe > it's essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > Best, > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > [1] https://github.com/ververica/flink-cdc-connectors > > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > [3] https://debezium.io > > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Thanks for all Flink CDC maintainers great work! Big +1. Best, Shengkai tison 于2023年12月7日周四 12:01写道： > This is very cool! +1 from my side. > > Best, > tison. > > Benchao Li 于2023年12月7日周四 11:56写道： > > > > Thank you, Leonard and all the Flink CDC maintainers. > > > > Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've > > already taken them as a whole project. > > > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > > > Dear Flink devs, > > > > > > As you may have heard, we at Alibaba (Ververica) are planning to > donate CDC Connectors for the Apache Flink project[1] to the Apache Flink > community. > > > > > > CDC Connectors for Apache Flink comprise a collection of source > connectors designed specifically for Apache Flink. These connectors[2] > enable the ingestion of changes from various databases using Change Data > Capture (CDC), most of these CDC connectors are powered by Debezium[3]. > They support both the DataStream API and the Table/SQL API, facilitating > the reading of database snapshots and continuous reading of transaction > logs with exactly-once processing, even in the event of failures. > > > > > > > > > Additionally, in the latest version 3.0, we have introduced many > long-awaited features. Starting from CDC version 3.0, we've built a > Streaming ELT Framework available for streaming data integration. This > framework allows users to write their data synchronization logic in a > simple YAML file, which will automatically be translated into a Flink > DataStreaming job. It emphasizes optimizing the task submission process and > offers advanced functionalities such as whole database synchronization, > merging sharded tables, and schema evolution[4]. > > > > > > > > > I believe this initiative is a perfect match for both sides. For the > Flink community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of > contributors. > > > > > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need > to be further discussed in this thread. For example, the sub-project name > after the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > maintainers are also highly active contributors to the Flink community, > donating this project to the Flink community implies that their permissions > might be reduced. Therefore, we may need to bring up this topic for further > discussion within the Flink PMC. Additionally, we need to discuss how to > migrate existing users and documents. We have a user group of nearly 10,000 > people and a multi-version documentation site need to migrate. We also need > to plan for the migration of CI/CD processes and other specifics. > > > > > > > > > While there are many intricate details that require implementation, we > are committed to progressing and finalizing this donation process. > > > > > > > > > Despite being Flink’s most active ecological project (as evaluated by > GitHub metrics), it also boasts a significant user base. However, I believe > it's essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > > > > > > > Really looking forward to hear what you think! > > > > > > > > > Best, > > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > > > [1] https://github.com/ververica/flink-cdc-connectors > > > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > > [3] https://debezium.io > > > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > > > > > > > -- > > > > Best, > > Benchao Li >
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
This is very cool! +1 from my side. Best, tison. Benchao Li 于2023年12月7日周四 11:56写道： > > Thank you, Leonard and all the Flink CDC maintainers. > > Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've > already taken them as a whole project. > > Leonard Xu 于2023年12月7日周四 11:25写道： > > > > Dear Flink devs, > > > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > > Connectors for the Apache Flink project[1] to the Apache Flink community. > > > > CDC Connectors for Apache Flink comprise a collection of source connectors > > designed specifically for Apache Flink. These connectors[2] enable the > > ingestion of changes from various databases using Change Data Capture > > (CDC), most of these CDC connectors are powered by Debezium[3]. They > > support both the DataStream API and the Table/SQL API, facilitating the > > reading of database snapshots and continuous reading of transaction logs > > with exactly-once processing, even in the event of failures. > > > > > > Additionally, in the latest version 3.0, we have introduced many > > long-awaited features. Starting from CDC version 3.0, we've built a > > Streaming ELT Framework available for streaming data integration. This > > framework allows users to write their data synchronization logic in a > > simple YAML file, which will automatically be translated into a Flink > > DataStreaming job. It emphasizes optimizing the task submission process and > > offers advanced functionalities such as whole database synchronization, > > merging sharded tables, and schema evolution[4]. > > > > > > I believe this initiative is a perfect match for both sides. For the Flink > > community, it presents an opportunity to enhance Flink's competitive > > advantage in streaming data integration, promoting the healthy growth and > > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > > becoming a sub-project of Apache Flink means being part of a neutral > > open-source community, which can attract a more diverse pool of > > contributors. > > > > Please note that the aforementioned points represent only some of our > > motivations and vision for this donation. Specific future operations need > > to be further discussed in this thread. For example, the sub-project name > > after the donation; we hope to name it Flink-CDC aiming to streaming data > > intergration through Apache Flink, following the naming convention of > > Flink-ML; And this project is managed by a total of 8 maintainers, > > including 3 Flink PMC members and 1 Flink Committer. The remaining 4 > > maintainers are also highly active contributors to the Flink community, > > donating this project to the Flink community implies that their permissions > > might be reduced. Therefore, we may need to bring up this topic for further > > discussion within the Flink PMC. Additionally, we need to discuss how to > > migrate existing users and documents. We have a user group of nearly 10,000 > > people and a multi-version documentation site need to migrate. We also need > > to plan for the migration of CI/CD processes and other specifics. > > > > > > While there are many intricate details that require implementation, we are > > committed to progressing and finalizing this donation process. > > > > > > Despite being Flink’s most active ecological project (as evaluated by > > GitHub metrics), it also boasts a significant user base. However, I believe > > it's essential to commence discussions on future operations only after the > > community reaches a consensus on whether they desire this donation. > > > > > > Really looking forward to hear what you think! > > > > > > Best, > > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > > > [1] https://github.com/ververica/flink-cdc-connectors > > [2] > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > > [3] https://debezium.io > > [4] > > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html > > > > -- > > Best, > Benchao Li
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Thank you, Leonard and all the Flink CDC maintainers. Big big +1 from me. As a heavy user of both Flink and Flink CDC, I've already taken them as a whole project. Leonard Xu 于2023年12月7日周四 11:25写道： > > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html -- Best, Benchao Li
Re: [PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Wow, Cool, Nice CDC is playing an increasingly important role. +1 Best, Jingsong On Thu, Dec 7, 2023 at 11:25 AM Leonard Xu wrote: > > Dear Flink devs, > > As you may have heard, we at Alibaba (Ververica) are planning to donate CDC > Connectors for the Apache Flink project[1] to the Apache Flink community. > > CDC Connectors for Apache Flink comprise a collection of source connectors > designed specifically for Apache Flink. These connectors[2] enable the > ingestion of changes from various databases using Change Data Capture (CDC), > most of these CDC connectors are powered by Debezium[3]. They support both > the DataStream API and the Table/SQL API, facilitating the reading of > database snapshots and continuous reading of transaction logs with > exactly-once processing, even in the event of failures. > > > Additionally, in the latest version 3.0, we have introduced many long-awaited > features. Starting from CDC version 3.0, we've built a Streaming ELT > Framework available for streaming data integration. This framework allows > users to write their data synchronization logic in a simple YAML file, which > will automatically be translated into a Flink DataStreaming job. It > emphasizes optimizing the task submission process and offers advanced > functionalities such as whole database synchronization, merging sharded > tables, and schema evolution[4]. > > > I believe this initiative is a perfect match for both sides. For the Flink > community, it presents an opportunity to enhance Flink's competitive > advantage in streaming data integration, promoting the healthy growth and > prosperity of the Apache Flink ecosystem. For the CDC Connectors project, > becoming a sub-project of Apache Flink means being part of a neutral > open-source community, which can attract a more diverse pool of contributors. > > Please note that the aforementioned points represent only some of our > motivations and vision for this donation. Specific future operations need to > be further discussed in this thread. For example, the sub-project name after > the donation; we hope to name it Flink-CDC aiming to streaming data > intergration through Apache Flink, following the naming convention of > Flink-ML; And this project is managed by a total of 8 maintainers, including > 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are > also highly active contributors to the Flink community, donating this project > to the Flink community implies that their permissions might be reduced. > Therefore, we may need to bring up this topic for further discussion within > the Flink PMC. Additionally, we need to discuss how to migrate existing users > and documents. We have a user group of nearly 10,000 people and a > multi-version documentation site need to migrate. We also need to plan for > the migration of CI/CD processes and other specifics. > > > While there are many intricate details that require implementation, we are > committed to progressing and finalizing this donation process. > > > Despite being Flink’s most active ecological project (as evaluated by GitHub > metrics), it also boasts a significant user base. However, I believe it's > essential to commence discussions on future operations only after the > community reaches a consensus on whether they desire this donation. > > > Really looking forward to hear what you think! > > > Best, > Leonard (on behalf of the Flink CDC Connectors project maintainers) > > [1] https://github.com/ververica/flink-cdc-connectors > [2] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html > [3] https://debezium.io > [4] > https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
[PROPOSAL] Contribute Flink CDC Connectors project to Apache Flink
Dear Flink devs, As you may have heard, we at Alibaba (Ververica) are planning to donate CDC Connectors for the Apache Flink project[1] to the Apache Flink community. CDC Connectors for Apache Flink comprise a collection of source connectors designed specifically for Apache Flink. These connectors[2] enable the ingestion of changes from various databases using Change Data Capture (CDC), most of these CDC connectors are powered by Debezium[3]. They support both the DataStream API and the Table/SQL API, facilitating the reading of database snapshots and continuous reading of transaction logs with exactly-once processing, even in the event of failures. Additionally, in the latest version 3.0, we have introduced many long-awaited features. Starting from CDC version 3.0, we've built a Streaming ELT Framework available for streaming data integration. This framework allows users to write their data synchronization logic in a simple YAML file, which will automatically be translated into a Flink DataStreaming job. It emphasizes optimizing the task submission process and offers advanced functionalities such as whole database synchronization, merging sharded tables, and schema evolution[4]. I believe this initiative is a perfect match for both sides. For the Flink community, it presents an opportunity to enhance Flink's competitive advantage in streaming data integration, promoting the healthy growth and prosperity of the Apache Flink ecosystem. For the CDC Connectors project, becoming a sub-project of Apache Flink means being part of a neutral open-source community, which can attract a more diverse pool of contributors. Please note that the aforementioned points represent only some of our motivations and vision for this donation. Specific future operations need to be further discussed in this thread. For example, the sub-project name after the donation; we hope to name it Flink-CDC aiming to streaming data intergration through Apache Flink, following the naming convention of Flink-ML; And this project is managed by a total of 8 maintainers, including 3 Flink PMC members and 1 Flink Committer. The remaining 4 maintainers are also highly active contributors to the Flink community, donating this project to the Flink community implies that their permissions might be reduced. Therefore, we may need to bring up this topic for further discussion within the Flink PMC. Additionally, we need to discuss how to migrate existing users and documents. We have a user group of nearly 10,000 people and a multi-version documentation site need to migrate. We also need to plan for the migration of CI/CD processes and other specifics. While there are many intricate details that require implementation, we are committed to progressing and finalizing this donation process. Despite being Flink’s most active ecological project (as evaluated by GitHub metrics), it also boasts a significant user base. However, I believe it's essential to commence discussions on future operations only after the community reaches a consensus on whether they desire this donation. Really looking forward to hear what you think! Best, Leonard (on behalf of the Flink CDC Connectors project maintainers) [1] https://github.com/ververica/flink-cdc-connectors [2] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-connectors.html [3] https://debezium.io [4] https://ververica.github.io/flink-cdc-connectors/master/content/overview/cdc-pipeline.html
