The zookeeper connections are persistent, so it depends on the number of clients more than the data flow rate on the producer side. If you are using a VIP based producer, then there is no connection from the producer process to zookeeper at all. If you are using a zookeper based producer, then you will have a connection per producer, and each of those connections will have several zookeeper watches on the broker nodes (so they know if one drops out).
On the consumer side, it is a little fuzzy... You will have a similar connections and set of watches on the brokers (again, so it can tell when they come and go). There will also be a set of watches and ephemeral nodes acting as locks on who is consuming from which broker-topic-partition. None of those are particularly heavy from the zookeeper service's perspective. Where you will see load is the offset commits - when the consumer writes the consumed offset to zookeeper for permanent storage (in case the consumer process dies or a rebalance causes the consumption ownership to change). Depending on the consumer processes' ability to handle duplicate data you want to set this to as long as is reasonable (if using the auto commit mechanism) or as large of a batch size as is reasonable (if using a manual commit mechanism). Here at LinkedIn, because of the overall number of consumers and topics being consumed, we're using a 60s commit interval in as many places as we can. -- Dave DeMaagd ddema...@linkedin.com | 818 262 7958 (sahmed1...@gmail.com - Mon, Dec 17, 2012 at 01:48:41PM -0500) > I was hoping someone could give a birds eye view of how zookeeper interacts > with kafka (server, producer, client). > > Once everything is running, how often do the various processes communicate > with zookeeper? > > If producer is sending 1000 messages per second, would it be 1000 > zookeeper connections per second (or more?)? > > On the consumer end, how often will it be connecting to zookeeper? I'm > sure it depends on the batch size and if you are part of a consumer group > etc.
