Re: Kafka consumer commit behaviour on rebalance

[Manoj Khangaonkar]

Yes. If a consumer when down, all the polled messages that
were not committed will be redelivered to another consumer.

regards



On Tue, Feb 13, 2018 at 9:31 AM, pradeep s 
wrote:

> Hi All,
> I am running a Kafka consumer(Single threaded) on kubernetes . Application
> is polling the records and accummulating in memory . There is a scheduled
> write of these records to S3 . Only after that i am committing the offsets
> back to Kafka.
> There are 4 partitions and 4 consumers(4 kubernetes pods) are there in the
> group.
>
> I have a question on the commit behaviour when kafka rebalancing happens .
> After a kafka rebalance , will the consumers get reassigned and get
> duplicate records .
> Do i need to clear my in memory buffer on a rebalance event
>
> Thanks
> Pradeep
>



-- 
http://khangaonkar.blogspot.com/

Kafka consumer commit behaviour on rebalance

[pradeep s]

Hi All,
I am running a Kafka consumer(Single threaded) on kubernetes . Application
is polling the records and accummulating in memory . There is a scheduled
write of these records to S3 . Only after that i am committing the offsets
back to Kafka.
There are 4 partitions and 4 consumers(4 kubernetes pods) are there in the
group.

I have a question on the commit behaviour when kafka rebalancing happens .
After a kafka rebalance , will the consumers get reassigned and get
duplicate records .
Do i need to clear my in memory buffer on a rebalance event

Thanks
Pradeep