correct, it has an ID
On Sun, Nov 25, 2012 at 10:48 PM, S Ahmed <sahmed1...@gmail.com> wrote: > How would you potentially discover a duplicate message, I guess your > message has a id/guid? > > > On Fri, Nov 16, 2012 at 8:50 PM, Joel Koshy <jjkosh...@gmail.com> wrote: > > > With compression enabled (as you have) it is possible for a consumer to > see > > duplicates during rebalance. This is because iteration may be in the > middle > > of a compressed message set just before a rebalance, but the checkpointed > > offsets are at MessageSet boundaries. However, this would only be during > > rebalance - i.e., in steady state, when you have no change in # > consumers/# > > partitions you shouldn't see duplicates. > > > > Joel > > > > > > On Fri, Nov 16, 2012 at 10:02 AM, Mark Grabois <mark.grab...@trendrr.com > > >wrote: > > > > > https://gist.github.com/4089354.git > > > > > > https://gist.github.com/4089369.git > > > > > > > > > > > > On Fri, Nov 16, 2012 at 12:59 PM, Mark Grabois < > mark.grab...@trendrr.com > > > >wrote: > > > > > > > Hi all, > > > > > > > > I'm encountering a problem where i'm getting far too many duplicate > > > > messages being sent to my kafka setup (statically, using > broker.list), > > > > being picked up by zk-based consumers. > > > > > > > > I've provided my test classes here and the kafka/zk versions i'm > using > > to > > > > run them and my servers: > > > > > > > > *client side*: > > > > producer: git://gist.github.com/4089354.git > > > > consumer: git://gist.github.com/4089369.git > > > > *jars*: > > > > kafka-0.7.2 > > > > > > > > *server side*: > > > > 5 kafka servers, zk servers on 3 of those > > > > 1 partition per test topic per server > > > > *server versions*: > > > > kafka-0.7.1 > > > > zookeeper-3.4.3 > > > > > > > > Any advice would be greatly appreciated. > > > > > > > > Thank you, > > > > Mark > > > > > > > > > > > > > > > > > >
