Consumer offsets in Zookeeper are not handled by the Kafka brokers at all - the consumer writes those directly to Zookeeper. Most likely, what you are seeing is the interval over which the consumer is committing offsets. Assuming that you are using the auto.commit.enable setting (it defaults to true, so unless you have changed that in the consumer configuration it will apply here), you should check the setting of auto.commit.interval for your consumer. I believe the default is still 60 seconds (or a config value of 60000). If you're seeing the values change in ZK every 5-6 seconds, then it sounds like it is set much lower.
The setting for the offset commit interval is entirely up to you. The lower the value, the more often offsets are checkpointed to Zookeeper. However that means that your consumer is writing to Zookeeper more often, and that has performance impacts. In an environment like ours, where there is a large number of consumers for a given cluster, we actually had to increase the interval to 10 minutes for quite some time because the load on Zookeeper was just too high. Once we moved the ZK transaction logs to SSDs, we were able to drop that back to 60 seconds. -Todd On Thu, Sep 17, 2015 at 1:14 PM, nitin sharma <kumarsharma.ni...@gmail.com> wrote: > anyone faced this issue? > > Regards, > Nitin Kumar Sharma. > > > On Wed, Sep 2, 2015 at 5:32 PM, nitin sharma <kumarsharma.ni...@gmail.com> > wrote: > > > Hi All, > > > > i have run into a weird issue with my Kafka setup. > > > > I see that it takes around 5-6 sec for Zookeeper to update the offset for > > Kafka topics. > > > > I am running "ConsumerOffsetChecker" tool to see that lag and what i > found > > that even when my consumer is not up, it takes 5-6 sec for Zookeeper to > > show the updated Offset . > > > > Is this behavior can be fixed? I have tried adding following parameters > in > > my Kafka server.property files but no luck. > > > > log.flush.interval.messages=1 > > log.default.flush.scheduler.interval.ms=500 > > > > > > ConsumerOffsetChecker: > > bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zkconnect > > localhost:2181 --group perfConsumer > > > > > > Regards, > > Nitin Kumar Sharma. > > > > >
