This sounds very useful! Another potential use case: - Consuming from multiple kafka clusters in different datacenters/regions.
I'm not sure if we would ultimately want to do this, but having it as an option when you need events from multiple kafka clusters to get the full history of changes (instead of relying on MirrorMaker) could be nice. On Mon, Jun 27, 2022 at 1:02 PM Ryan van Huuksloot < ryan.vanhuuksl...@shopify.com> wrote: > Hi Mason, > > Thanks for starting this discussion! The proposed Source sounds awesome > and we would be interested in taking a look at the source code and > evaluating our use cases. We can provide information and review on a > potential FLIP based on other use cases. > > Do you have a fork/branch that you are working with that is public? Could > you attach that so we can start looking at it? > > Let us know if you need anything from us to help move this forward. > > Thanks! > Ryan > > On 2022/06/27 03:08:13 Qingsheng Ren wrote: > > Hi Mason, > > > > It sounds like an exciting enhancement to the Kafka source and will > benefit a lot of users I believe. > > > > Would you prefer to reuse the existing flink-connector-kafka module or > create a new one for the new multi-cluster feature? Personally I prefer the > former one because users won’t need to introduce another dependency module > to their projects in order to use the feature. > > > > Thanks for the effort on this and looking forward to your FLIP! > > > > Best, > > Qingsheng > > > > > On Jun 24, 2022, at 09:43, Mason Chen <ma...@gmail.com> wrote: > > > > > > Hi community, > > > > > > We have been working on a Multi Cluster Kafka Source and are looking to > > > contribute it upstream. I've given a talk about the features and > design at > > > a Flink meetup: https://youtu.be/H1SYOuLcUTI. > > > > > > The main features that it provides is: > > > 1. Reading multiple Kafka clusters within a single source. > > > 2. Adjusting the clusters and topics the source consumes from > dynamically, > > > without Flink job restart. > > > > > > Some of the challenging use cases that these features solve are: > > > 1. Transparent Kafka cluster migration without Flink job restart. > > > 2. Transparent Kafka topic migration without Flink job restart. > > > 3. Direct integration with Hybrid Source. > > > > > > In addition, this is designed with wrapping and managing the existing > > > KafkaSource components to enable these features, so it can continue to > > > benefit from KafkaSource improvements and bug fixes. It can be > considered > > > as a form of a composite source. > > > > > > I think the contribution of this source could benefit a lot of users > who > > > have asked in the mailing list about Flink handling Kafka migrations > and > > > removing topics in the past. I would love to hear and address your > thoughts > > > and feedback, and if possible drive a FLIP! > > > > > > Best, > > > Mason > > > > >
