From: Ninad Naik <<>>
Date: Friday, August 4, 2017 at 1:53 PM
To: "<>" 
Subject: Kafka (0.9) ConsumerRebalanceListener onPartitionsRevoked not called.

We’re using custom storage for kafka offsets. Partitions are assigned 
automatically, and we’re using ConsumerRebalanceListener to save offsets, clear 
threads and so on. In ‘onPartitionsAssinged’ method, we look up committed 
offsets for assigned partitions from the custom store. If no committed offsets 
are found, we get appropriate offsets (earliest/latest) from kafka.

According to javadocs for ConsumerRebalanceListener:

"It is guaranteed that all consumer processes will invoke 
 prior to any process invoking 

However, we have seen this a couple of times now, that on a re-balance, 
‘onPartitionsRevoked’ isn’t called. ‘onPartitionsAssigned’ gets called 
directly, which gets our application in a bad state.

We looked briefly at the code, and seems like the code is doing as promised by 
the javadocs. So, the question is, is there any edge case, any flow out there 
which can possibly trigger this?

One thing we observed, in both cases, was that while trying to get offsets from 
kafka (because offsets were not present in custom store), Fetcher retried at 
least once due to obsolete leadership information:

Attempt to fetch offsets for partition {} failed due to obsolete leadership 
information, retrying.

Please let us know.

Ninad Naik.

Reply via email to