In trunk, the # of dups introduced during rebalance is significantly reduced. We used to replay the last chunk of fetched messages during rebalance. In trunk, there is at most 1 duplicated message per partition during rebalance (assuming messages are not compressed).
Jun On Wed, May 16, 2012 at 2:41 AM, navneet sharma <navneetsharma0...@gmail.com > wrote: > Hi, > > I tried a scenario wherein: > 1) i had 1 producer and 3 consumers subscribed for a topic - "cartTopic", > all in same group. > 2) Now, when everything is executing, i introduced another consumer for the > same topic and in the same group. So, overall there are 4 consumers. > 3) Ofcourse, it triggered re-balancing. > > But then final result is that few messages are duplicated. > In my example run, producer sent 800,000 records, but consumer received > 801,448 records. > I am using log4j to generate the output file. > > Is there any reasons for duplicacy? > > Thanks, > Navneet Sharma >
