The existing consumer uses Zookeeper both to commit offsets and to assign partitions to different consumers and threads in the same consumer group. While offsets can be committed to Kafka in 0.8.2 releases, which greatly reduces the load on Zookeeper, the consumer still requires Zookeeper to manage group membership and partition ownership. The new consumer (which we hope to have ready for 0.8.3 release) will completely remove the Zookeeper dependency, managing both offsets and partition ownership within Kafka itself.
Gwen On Tue, Jun 9, 2015 at 10:26 AM, noah <iamn...@gmail.com> wrote: > We are setting up a new Kafka project (0.8.2.1) and are trying to go > straight to consumer offsets stored in Kafka. Unfortunately it looks like > the Java consumer will try to connect to ZooKeeper regardless of the > settings. > > Will/When will this dependency go away completely? It would simplify our > deployments if our consumers didn't have to connect to ZooKeeper at all. > > P.S. I've asked this on Stack Overflow, if you would like to answer there > for posterity: > http://stackoverflow.com/questions/30719331/kafka-0-8-2-1-offsets-storage-kafka-still-requires-zookeeper
