Yes, which is why I asked how discovery will work. Proposal II reads a bit handwavy.
"On startup, the consumer is informed with a list of the brokers." >From where? From whom? I have to provide some mechanism now to feed this config to my consumers? What happens when this changes? Say I start up my producers and consumers with a static list of hosts A,B,C. Over time, I add host D, E, F. D becomes the eventual new coordinator. But the clients have an in-memory config which knows only of A, B, and C. Now host D dies. What happens (especially to clients who have never been told about/connected to E or F)? Also is this proposal going to address the ability to drain a broker? Say I have a broker with old data that I want to keep around so you can reconsume old data, but I don't want anyone producing to it anymore. --Dave On Wed, Jun 20, 2012 at 7:48 AM, Jun Rao <jun...@gmail.com> wrote: > Dave, > > Just to clarify. We are not removing ZK completely. The proposal is just > trying to see if we can remove the ZK dependency on the consumer client. ZK > will still be used at the broker. > > Thanks, > > Jun > > On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <dave.b...@gmail.com> wrote: > >> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <neha.narkh...@gmail.com> >> wrote: >> > One of the goals of thinning the Kafka consumer client is removing the >> > zookeeper client from the consumer. Without this, Kafka consumer >> > client would depend on the stability of a zookeeper client. >> >> If there's a stability issue with the zookeeper client, then that >> should be addressed. >> >> ZK is a fine tool for service discovery and coordination. It seems >> like any new system that forced me, as a consumer, to use yet another >> system to bootstrap and discover where my brokers are for a topic >> would be a step backward. >> >> I'm curious, why, specifically, is removing ZK a design goal >> (especially when it's such a core component of the broker)? I think >> of other projects, like HBase, which seem to have no issue with using >> ZK in their client. >> >> --Dave >>
