Sorry, but all of this I know and it doesn't answer the question. The
question I have is assume that I have 3 consumers (think KafkaConsumer java
class):
Consumer 1 assigned to partitions {0, 1}
Consumer 2 assigned to partitions {2, 3}
Consumer 3 assigned to partitions {4, 5}
Now assume consumer 1 dies or leaves the consumer group whatever.
Partitions are then rebalanced and you could only move 0, 1 to the existing
consumers:
Consumer 2 assigned to partitions {0, 2, 3}
Consumer 3 assigned to partitions {1, 4, 5}
However, you could also just shuffle the set {0, 1, 2, 3, 4, 5} and
randomly assign them e.g. :
Consumer 2 assigned to partitions {1, 2, 5}
Consumer 3 assigned to partitions {0, 3, 4}
In the former case, only two partitions were moved to a new consumer, but
in the latter, 4 partitions were moved.
Does Kafka strive to do the former? Is there some form of algorithmic
guarantee for how the new partition assignment is computed, so I can build
a deployment strategy around it?
Best,
Edvard
On Thu, Jun 8, 2023 at 1:53 PM sunil chaudhari <sunilmchaudhar...@gmail.com>
wrote:
> I will try to answer.
> rebalancing triggers when one or two consuemrs(client) leaves the group
> because of any reason.
> The thumb rule is Number of partitions should be equal to number of
> consumer threads.
> If there are 300 partitions assigned one thread each it wont rebalance
> untill some consumer marked as dead.
> How it marks as dead: if the kafka doesnt receive heartbeat from consumer
> in 5 mins(or defined at client side).
>
> If 20 consumers are dead from kafka persepctive at time T1 then it will
> trigger rebalance.
> It will trigger rebalance at time T2 when there is new consumer added to
> the group and there is poll request from new consumer.
>
> If there is no issue with number of partitions and number of consumers then
> it wont trigger rebalance.
>
> Terms I used may not be accurate😊
>
> Regards,
> Sunil.
>
> On Thu, 8 Jun 2023 at 3:59 PM, Edvard Fagerholm <
> edvard.fagerh...@gmail.com>
> wrote:
>
> > Hello,
> >
> > I couldn't find an answer in the documentation to the following. If a new
> > machine joins a consumer group and Kafka triggers a rebalance, will it
> > randomly reassign partitions or will it hand over partitions from
> existing
> > consumers to the newly joined one? In other words, will it attempt to
> move
> > as few partitions as possible between consumers?
> >
> > The main implications of this is local in-memory caches and scaling up
> the
> > number of machines in a consumer group, since the scaling up operation
> will
> > require nuking any local caches for the partitions that were moved. This
> > would cause a spike on any DBs that are being cached on the consumers.
> >
> > Best,
> > Edvard
> >
>
