Thanks for clarifying Mathieu, I hadn't seen your PR yet and assumed from your earlier comments that this was still a WIP.
I do tend to agree with David on this though, I was also struggling to understand the use case/motivation and felt it was quite specific. As I think David said in his PR comment, we've tried to limit the assignors we include to more common and simple use cases that are easy for people to understand and choose between. Of course since there is now a PR for it, if any others do have a similar use case and would find it helpful, they can plug in a custom assignor based on your patch. If we do see this happening often enough that people start requesting it be added to the OOTB assignors, then it would make sense to revisit this PR. As it is, I suspect just putting it out there for others to use via the pluggable interface may be helpful enough. Either way, thanks for the contribution! -Sophie On Mon, Oct 24, 2022 at 11:19 PM David Jacot <david.ja...@gmail.com> wrote: > Hi Mathieu, > > Thanks for the effort that you have put in creating this KIP. I just read > it again and I am still confused by the use cases and the motivation. I > suppose that this works for your data model but it does not seem to be a > general pattern. > > Overall, I stick to the comment that I made in the PR. I still feel like > that this is something too use case specific to make it into Apache Kafka. > > Best, > David > > Le mar. 25 oct. 2022 à 07:25, Mathieu Amblard <mathieu.ambl...@gmail.com> > a > écrit : > > > Hi Sophie, > > > > Thank you for your message. > > > > The TopicRoundRobinAssignor has been already implemented (because the > > existing ones do not address our use cases correctly), it implements the > > interface you mentionned, and I have made a PR to the Kafka repo where > you > > can see the source code. It was in a comment of that PR that someone from > > the core team proposes to create a KIP. > > > > I have also fully unit tested this assignor (see the PR). Moreover, > around > > 20 microservices are using it in production, for 6 months now, in the > > company I am working for. So I think it has been proven and that's why I > > have made this proposal. I would never have dared to suggest something I > > have never tested and if there is no need for it. > > > > Finally, this assignor suits well our Kafka architecture and use cases > > contrary to the existing ones. For reminder, we are using Kafka as a > > messaging system (no streaming) and we can have only one type of message > > per topic. Maybe it is too specific, maybe not, that's also why I have > made > > this KIP, to challenge it, to see if someone has the same needs and if > this > > assignor can help others. If not, that's not a problem, I will simply > keep > > it in our source code. > > > > Regards, > > Mathieu > > > > > > Le mar. 25 oct. 2022 à 01:51, Sophie Blee-Goldman > > <sop...@confluent.io.invalid> a écrit : > > > > > Hey Mathieu, > > > > > > Apologies if you already know this, but the partition assignor > interface > > is > > > fully pluggable. > > > That means you can plug in a custom PartitionAssignor implementation, > > > without > > > having to go through a KIP or commit any code to the AK repo. > > > > > > I suspect some of the current unknowns and solutions will become clear > > when > > > you start > > > to actually write this assignor and test it out in your environment. > Then > > > you can play around > > > with what works and what doesn't work, and come back to the KIP if > > desired > > > with a stronger > > > argument for why it's needed. Or you can just take your assignor and > > > publish it in a public > > > git repo for anyone who might have a similar use case as you. > > > > > > Just my two cents, I'd recommend in this case you start with the > > > implementation before > > > worrying about donating your assignor back to the main repo with a KIP. > > IF > > > you do want to, > > > it would then be much easier to convince people when they can see your > > > assignor logic > > > for themselves, and you'll be able to answer any questions. > > > > > > Best, > > > Sophie > > > > > > On Fri, Oct 21, 2022 at 2:21 AM Mathieu Amblard < > > mathieu.ambl...@gmail.com > > > > > > > wrote: > > > > > > > Hello everybody, > > > > > > > > Just to let you know that I have added a chapter about having > multiple > > > > containers (multiple pods for Kubernetes) running the same > application > > : > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-874%3A+TopicRoundRobinAssignor#KIP874:TopicRoundRobinAssignor-Howdoesitworkifwehavemultiplecontainersrunningthesameapplication > > > > ? > > > > > > > > Regards, > > > > Mathieu > > > > > > > > Le mar. 11 oct. 2022 à 20:00, Mathieu Amblard < > > mathieu.ambl...@gmail.com > > > > > > > > a > > > > écrit : > > > > > > > > > Hi Hector, > > > > > > > > > > > > > > > > > > > > First, thank you for your questions ! > > > > > > > > > > > > > > > > > > > > *If the goal is to do the partition assignments at a topic level, > > > > wouldn't > > > > > having single-partition topics solve this problem?* > > > > > > > > > > > > > > > > > > > > We are in a microservices environment; therefore, we can have > > multiple > > > > > containers running the same application. > > > > > > > > > > Using the TopicRoundRobinAssignor, the partitions are uniformly > > > balanced > > > > > to each container, and we can get better performances. > > > > > > > > > > Let suppose there are 2 instances of the same application A0 and > A1, > > 2 > > > > > consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions > > and > > > t1 > > > > > has two partitions resulting in partitions : t0p0, t0p1, t0p2, > t1p0, > > > > t1p1. > > > > > > > > > > If we use the TopicRoundRobinAssignor, the assignment will be : > > > > > > > > > > A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] > > > > > > > > > > A1 : [ C0: [t0p1], C1: [t1p1] ] > > > > > > > > > > > > > > > > > > > > *How will the group leader know that T2 should not be re-assigned > on > > > the > > > > > next rebalance? Can you elaborate a bit more on the mechanisms used > > to > > > > > communicate this state to the other group members?* > > > > > > > > > > > > > > > > > > > > Currently, the group leader will not know that T2 should not be > > > > > re-assigned on the next balance. > > > > > > > > > > For this first iteration, we simply keep in memory that T2 has a > > poison > > > > > pill and therefore we ignore all incoming messages from T2. We > > > basically > > > > > consume them without acknowledging them. > > > > > > > > > > As you can imagine, in the case of having multiple instances of the > > > same > > > > > application, in case of error, the partition will be rebalanced to > > > > another > > > > > instance. > > > > > > > > > > Nevertheless, this is not really a problem (at least for our use > > > cases), > > > > > as soon as the poison pill is consumed, the consumer of this other > > > > instance > > > > > will be stopped, and so on, and so on. It will take a few tens of > > > seconds > > > > > before the last consumer of the poison pill will be stopped and so > > the > > > > > consumption of the entire topic. > > > > > > > > > > For a second iteration, we have planned to find a solution to avoid > > > this > > > > > time lapse between the consumer of the first instance being stopped > > and > > > > the > > > > > last one. Currently, I do not have a solution, but we are thinking > > > about > > > > > different options, avoiding rebalancing partitions containing a > > poison > > > > pill > > > > > is one of them. > > > > > > > > > > > > > > > > > > > > Cheers, > > > > > > > > > > Mathieu > > > > > > > > > > Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD > A) > > < > > > > > hgerald...@bloomberg.net> a écrit : > > > > > > > > > >> Hi Mathieu. I took a look at your KIP and have a couple questions. > > > > >> > > > > >> If the goal is to do the partition assignments at a topic level, > > > > wouldn't > > > > >> having single-partition topics solve this problem? > > > > >> > > > > >> You also mentioned that your goal is to minimize the potential of > a > > > > >> poison pill message breaking all members of a group (by keeping > > track > > > of > > > > >> which topics have 'failed'), but it is not clear how this can be > > > > achieved > > > > >> with this assignor. If we imagine an scenario where: > > > > >> > > > > >> * A group has 3 members (A, B, C) > > > > >> * Members are subscribed to 3 topics (T1, T2, T3) > > > > >> * Each member is assigned one topic (A[T1], B[T2], C[T3]) > > > > >> * One member fails to consume from a topic/partition (B[T2]), and > > goes > > > > >> into failed state > > > > >> > > > > >> How will the group leader know that T2 should not be re-assigned > on > > > the > > > > >> next rebalance? Can you elaborate a bit more on the mechanisms > used > > to > > > > >> communicate this state to the other group members? > > > > >> > > > > >> Thanks > > > > >> > > > > >> From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > > > > >> dev@kafka.apache.org > > > > >> Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > > > > >> > > > > >> Hi Kafka Developers, > > > > >> > > > > >> My proposal is to add a new partition assignment strategy at the > > topic > > > > >> level to : > > > > >> - have a better data consistency by consumed topic in case of > > > exception > > > > >> - have a solution much thread safe for the consumer > > > > >> In case there are multiple consumers and multiple topics. > > > > >> > > > > >> Here is the link to the KIP with all the explanations : > > > > >> https://cwiki.apache.org/confluence/x/XozGDQ > > > > >> > > > > >> Thank you in advance for your feedbacks, > > > > >> Mathieu > > > > >> > > > > >> > > > > >> > > > > > > > > > >
