Hi, I am in the middle of writing a new KIP to introduced a new interface for topic updates/creation/description that will allow MM2 to either uses the default behaviour which uses Kafka AdminClient to create/update topics or to use a customised one that integrates with the user's ecosystem where they can call their Kafka resource management service to create a topic or update the configuration of the topic. Would this be a possible solution for your use case where you can provide a custom implementation that works with your ecosystem to create the topic?
-- Omnia On Mon, Jun 14, 2021 at 2:25 PM Igor Soarez <i...@soarez.me> wrote: > Maybe it would be nice if it was possible to hook into or extend Admin > client interactions, to allow for custom logic supporting use cases such as > this. > Scenarios where topic/resource management is centralized are probably not > uncommon. > > -- > Igor > > On Sat, Jun 12, 2021, at 9:42 AM, Matthew de Detrich wrote: > > Thanks Ryanne and Mickael > > > > I already suspected that something like this (i.e. current behaivior > being > > a deliberate design decision) was the case and I just wanted to confirm > my > > suspicions. > > > > I will relay this internally > > > > On Fri, 11 Jun 2021, 18:55 Mickael Maison, <mickael.mai...@gmail.com> > wrote: > > > > > Hi Matthew, > > > > > > If an administrator want to control topic creation, they should use > > > ACLs to prevents users creating topic. Relying on all applications to > > > not create topics is unlikely to succeed. > > > > > > Each refresh.topics.interval.seconds, MM2 checks if it needs to create > > > topics/partitions by comparing both clusters and also their previous > > > states. If your automation is able to create new topics on both > > > clusters, MM2 should detect them correctly and not attempt creating > > > any topics. If a topic does not exist on the target cluster, MM2 will > > > try creating it. If it fails, it will retry again at the next > > > refresh.topics.interval.seconds until the topic gets created. However, > > > it will also trigger a task reconfiguration each time which may have > > > an impact on your mirroring throughput. > > > > > > On Fri, Jun 11, 2021 at 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> > > > wrote: > > > > > > > > Matthew, I wonder what the expected behavior would be when > topic-creation > > > > is disabled and MM is asked to replicate a topic that doesn't exist > on > > > the > > > > target cluster? Maybe the task should fail at that point, or maybe it > > > > should replicate whatever it can? > > > > > > > > I think the current behavior is reasonable, esp considering precedent > > > from > > > > Connect and Streams, both of which actively create topics as needed. > > > > > > > > But I understand the motivation. Have they considered revoking topic > > > > creation permission using ACLs? > > > > > > > > Ryanne > > > > > > > > On Fri, Jun 11, 2021, 3:54 AM Matthew de Detrich > > > > <matthew.dedetr...@aiven.io.invalid> wrote: > > > > > > > > > Hello everyone, > > > > > > > > > > We have an interesting feature request from a client regarding > having > > > the > > > > > property of automatic topic creation to be reflected in a MM2. > > > Specifically > > > > > the current behaviour where if you have automatic topic creation > set to > > > > > false for the original Kafla cluster, MM2 configuration ignores > this > > > which > > > > > means that if Kafka clients send messages to the MM2 then topics > will > > > still > > > > > be automatically created on target cluster. The core problem here > for > > > the > > > > > client is that our client wants to have complete control over how > > > topics > > > > > get created (i.e. with terraform setup scripts) and with the > current > > > > > behaviour this is not possible. > > > > > > > > > > Of course this poses other problems if we did want to change the > > > behaviour > > > > > as stated earlier, i.e. if you update the configuration for the > > > original > > > > > Kafka cluster then you get into open questions about how to reflect > > > this > > > > > configuration onto the mirror maket (this is why its called > "mirror"). > > > Is > > > > > making MM2 reflect that flag a feature that makes general or > > > alternately is > > > > > there another variation that makes more sense (i.e. having a > separate > > > > > specific property rather than reusing the current automatic topic > > > creation > > > > > one). > > > > > > > > > > There is a currently existing issue on this at > > > > > https://issues.apache.org/jira/browse/KAFKA-12753 > > > > > > > > > > @Ryanne @Mickael Since you guys are the main developers on MM/MM2 > what > > > are > > > > > your thoughts on this? > > > > > > > > > > > > > > > -- > > > > > > > > > > Matthew de Detrich > > > > > > > > > > *Aiven Deutschland GmbH* > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > > > > > *m:* +491603708037 > > > > > > > > > > *w:* aiven.io *e:* matthew.dedetr...@aiven.io > > > > > > > > > > >
