This will force a rewrite of those monitoring tools and UI tools that read offsets from Zookeeper in order directly to get lag information for reporting on consumer clients. It seems a good thing to know this is coming down the pipe.
Dennis On Wed, Jun 4, 2014 at 6:50 PM, Neha Narkhede <neha.narkh...@gmail.com> wrote: > At LinkedIn, we are just beginning to move our applications to Kafka based > offset management. I'm not sure if Todd/Clark/Joel have any numbers to > share right away, but depending on the number of consumers you have and the > number of topics those consumers pull, it will likely make a sizable impact > on the performance of your zookeeper cluster. It will not, however, > influence the decision of sizing the zookeeper cluster since zookeeper > writes do not scale with the number of servers in a zookeeper cluster. In > general, it is recommended to keep a zookeeper cluster small, of the size > of 3 or 5 servers. i.e. whether or not your consumers checkpoint offsets to > zookeeper or Kafka. It is true that the writes will be faster on a 3 node > zookeeper cluster vs a 5 node zookeeper cluster. > > Thanks > Neha > > > On Wed, Jun 4, 2014 at 11:15 AM, S Ahmed <sahmed1...@gmail.com> wrote: > > > Very nice. > > > > Do you guys have any stats on what kind of load was reduced on ZK? Just > > trying to understand if this changes the type of servers required to host > > ZK. > > > > > > > > > > On Wed, Jun 4, 2014 at 1:10 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Yes, we are migrating the offset management from ZK to the broker as a > > > special log. > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/Inbuilt+Consumer+Offset+Management > > > > > > The code is in trunk, and it is running in production at LinkedIn now. > > > > > > Guozhang > > > > > > > > > On Wed, Jun 4, 2014 at 10:00 AM, S Ahmed <sahmed1...@gmail.com> wrote: > > > > > > > I swear I read that Jay Kreps wrote somewhere that consumers now > write > > > > their offsets in a logfile (not in zookeeper). > > > > > > > > Is this true or did I misread? Sorry I can't find the article I was > > > > reading. > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > >
