Re: How Kafka handle partition leader change?

2023-11-22 Thread Jack Vanlightly
> Thanks Andrew. > > From: Andrew Grant > Sent: 21 November 2023 12:35 > To: dev@kafka.apache.org > Subject: Re: How Kafka handle partition leader change? > > Hey De Gao, > > Message loss or duplication can actually happen even without a leadership > change for a partit

Re: How Kafka handle partition leader change?

2023-11-21 Thread De Gao
is not about fix the message loss. Will share when ready. Thanks Andrew. From: Andrew Grant Sent: 21 November 2023 12:35 To: dev@kafka.apache.org Subject: Re: How Kafka handle partition leader change? Hey De Gao, Message loss or duplication can actually happen

Re: How Kafka handle partition leader change?

2023-11-21 Thread Andrew Grant
Gao > Sent: 21 November 2023 6:25 > To: dev@kafka.apache.org > Subject: Re: How Kafka handle partition leader change? > > Thanks Andrew. Sounds like the leadership change from Kafka side is a 'best > effort' to avoid message duplicate or loss. Can we say that message lo

Re: How Kafka handle partition leader change?

2023-11-20 Thread De Gao
To: dev@kafka.apache.org Subject: Re: How Kafka handle partition leader change? Thanks Andrew. Sounds like the leadership change from Kafka side is a 'best effort' to avoid message duplicate or loss. Can we say that message lost is very likely during leadership change unless pro

Re: How Kafka handle partition leader change?

2023-11-20 Thread De Gao
ntegration guarantee upon metadata change? From: Andrew Grant Sent: 20 November 2023 12:26 To: dev@kafka.apache.org Subject: Re: How Kafka handle partition leader change? Hey De Gao, The controller is the one that always elects a new leader. When that hap

Re: How Kafka handle partition leader change?

2023-11-20 Thread Andrew Grant
Hey De Gao, The controller is the one that always elects a new leader. When that happens that metadata is changed on the controller and once committed it’s broadcast to all brokers in the cluster. In KRaft this would be via a PartitonChange record that each broker will fetch from the controller

How Kafka handle partition leader change?

2023-11-19 Thread De Gao
Hi all I have a interesting question here. Let's say we have 2 broker B1 B2, controller C and producer P1, P2...Pn. Currently B1 holds the partition leader and Px is constantly producing messages to B1. We want to move the partition leadership to B2. How does the leadership change synced betwee