The proposal is to move the rebalancing logic out of the consumer to the broker side. The consumer client merely connects to the consumer co-ordinator (on the Kafka brokers) and fetches data from the assigned set of partitions. We have a high level design here - https://cwiki.apache.org/confluence/display/KAFKA/Central+Consumer+Coordination#CentralConsumerCoordination-ProposalII
Thanks, Neha On Wed, Jun 20, 2012 at 9:36 AM, Evan Chan <e...@ooyala.com> wrote: > How would you accomplish automatic rebalancing without ZK? > > On Wed, Jun 20, 2012 at 8:25 AM, Taylor Gautier <tgaut...@tagged.com> wrote: > >> Fwiw I think this is the right move. We don't use ZK in our Kafka >> installation. >> >> One reason is that we do not want or need the complexity of having the >> broker distribution in the clients. >> >> Moving towards this design in the core would help our installation be >> more "standard". >> >> >> >> On Jun 20, 2012, at 7:48 AM, Jun Rao <jun...@gmail.com> wrote: >> >> > Dave, >> > >> > Just to clarify. We are not removing ZK completely. The proposal is just >> > trying to see if we can remove the ZK dependency on the consumer client. >> ZK >> > will still be used at the broker. >> > >> > Thanks, >> > >> > Jun >> > >> > On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <dave.b...@gmail.com> wrote: >> > >> >> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede < >> neha.narkh...@gmail.com> >> >> wrote: >> >>> One of the goals of thinning the Kafka consumer client is removing the >> >>> zookeeper client from the consumer. Without this, Kafka consumer >> >>> client would depend on the stability of a zookeeper client. >> >> >> >> If there's a stability issue with the zookeeper client, then that >> >> should be addressed. >> >> >> >> ZK is a fine tool for service discovery and coordination. It seems >> >> like any new system that forced me, as a consumer, to use yet another >> >> system to bootstrap and discover where my brokers are for a topic >> >> would be a step backward. >> >> >> >> I'm curious, why, specifically, is removing ZK a design goal >> >> (especially when it's such a core component of the broker)? I think >> >> of other projects, like HBase, which seem to have no issue with using >> >> ZK in their client. >> >> >> >> --Dave >> >> >> > > > > -- > -- > *Evan Chan* > Senior Software Engineer | > e...@ooyala.com | (650) 996-4600 > www.ooyala.com | blog <http://www.ooyala.com/blog> | > @ooyala<http://www.twitter.com/ooyala>
