Hi Hector,
First, thank you for your questions ! *If the goal is to do the partition assignments at a topic level, wouldn't having single-partition topics solve this problem?* We are in a microservices environment; therefore, we can have multiple containers running the same application. Using the TopicRoundRobinAssignor, the partitions are uniformly balanced to each container, and we can get better performances. Let suppose there are 2 instances of the same application A0 and A1, 2 consumers C0 and C1, and two topics t0 and t1. t0 has 3 partitions and t1 has two partitions resulting in partitions : t0p0, t0p1, t0p2, t1p0, t1p1. If we use the TopicRoundRobinAssignor, the assignment will be : A0 : [ C0: [t0p0, t0p2], C1: [t1p0] ] A1 : [ C0: [t0p1], C1: [t1p1] ] *How will the group leader know that T2 should not be re-assigned on the next rebalance? Can you elaborate a bit more on the mechanisms used to communicate this state to the other group members?* Currently, the group leader will not know that T2 should not be re-assigned on the next balance. For this first iteration, we simply keep in memory that T2 has a poison pill and therefore we ignore all incoming messages from T2. We basically consume them without acknowledging them. As you can imagine, in the case of having multiple instances of the same application, in case of error, the partition will be rebalanced to another instance. Nevertheless, this is not really a problem (at least for our use cases), as soon as the poison pill is consumed, the consumer of this other instance will be stopped, and so on, and so on. It will take a few tens of seconds before the last consumer of the poison pill will be stopped and so the consumption of the entire topic. For a second iteration, we have planned to find a solution to avoid this time lapse between the consumer of the first instance being stopped and the last one. Currently, I do not have a solution, but we are thinking about different options, avoiding rebalancing partitions containing a poison pill is one of them. Cheers, Mathieu Le ven. 7 oct. 2022 à 16:54, Hector Geraldino (BLOOMBERG/ 919 3RD A) < hgerald...@bloomberg.net> a écrit : > Hi Mathieu. I took a look at your KIP and have a couple questions. > > If the goal is to do the partition assignments at a topic level, wouldn't > having single-partition topics solve this problem? > > You also mentioned that your goal is to minimize the potential of a poison > pill message breaking all members of a group (by keeping track of which > topics have 'failed'), but it is not clear how this can be achieved with > this assignor. If we imagine an scenario where: > > * A group has 3 members (A, B, C) > * Members are subscribed to 3 topics (T1, T2, T3) > * Each member is assigned one topic (A[T1], B[T2], C[T3]) > * One member fails to consume from a topic/partition (B[T2]), and goes > into failed state > > How will the group leader know that T2 should not be re-assigned on the > next rebalance? Can you elaborate a bit more on the mechanisms used to > communicate this state to the other group members? > > Thanks > > From: dev@kafka.apache.org At: 10/05/22 03:47:33 UTC-4:00To: > dev@kafka.apache.org > Subject: [DISCUSS] KIP-874: TopicRoundRobinAssignor > > Hi Kafka Developers, > > My proposal is to add a new partition assignment strategy at the topic > level to : > - have a better data consistency by consumed topic in case of exception > - have a solution much thread safe for the consumer > In case there are multiple consumers and multiple topics. > > Here is the link to the KIP with all the explanations : > https://cwiki.apache.org/confluence/x/XozGDQ > > Thank you in advance for your feedbacks, > Mathieu > > >
