Dave, > Say I start up my producers and consumers with a static list of hosts > A,B,C. Over time, I add host D, E, F. D becomes the eventual new > coordinator. But the clients have an in-memory config which knows > only of A, B, and C. Now host D dies. What happens (especially to > clients who have never been told about/connected to E or F)?
The consumers would need only one of the brokers in its config to be alive. It talks to one of the live brokers in the boostrap brokers list and finds out who the co-ordinator is, and then communicates with the co-ordinator. So, in your example, unless you take A, B AND C offline, it shouldn't be a problem. And if you need to do that, you can rolling restart the consumers. Maybe there is another easier way to solve the bootstrap problem, this is just one of the solutions. >> > Also is this proposal going to address the ability to drain a broker? > Say I have a broker with old data that I want to keep around so you > can reconsume old data, but I don't want anyone producing to it > anymore. This proposal is only dealing with Consumer API/protocol redesign. What you want is the ability to decommission a broker gracefully. I suggest you add your comments/requirements to this JIRA - https://issues.apache.org/jira/browse/KAFKA-155 Also, how about moving your suggestions/comments to the JIRA/wiki ? Email is not quite the easiest way to keep track of this discussion. Thanks, Neha On Wed, Jun 20, 2012 at 12:11 PM, Dave Barr <dave.b...@gmail.com> wrote: > Yes, which is why I asked how discovery will work. > > Proposal II reads a bit handwavy. > > "On startup, the consumer is informed with a list of the brokers." > > From where? From whom? I have to provide some mechanism now to feed > this config to my consumers? What happens when this changes? > > Say I start up my producers and consumers with a static list of hosts > A,B,C. Over time, I add host D, E, F. D becomes the eventual new > coordinator. But the clients have an in-memory config which knows > only of A, B, and C. Now host D dies. What happens (especially to > clients who have never been told about/connected to E or F)? > > Also is this proposal going to address the ability to drain a broker? > Say I have a broker with old data that I want to keep around so you > can reconsume old data, but I don't want anyone producing to it > anymore. > > --Dave > On Wed, Jun 20, 2012 at 7:48 AM, Jun Rao <jun...@gmail.com> wrote: >> Dave, >> >> Just to clarify. We are not removing ZK completely. The proposal is just >> trying to see if we can remove the ZK dependency on the consumer client. ZK >> will still be used at the broker. >> >> Thanks, >> >> Jun >> >> On Tue, Jun 19, 2012 at 10:49 PM, Dave Barr <dave.b...@gmail.com> wrote: >> >>> On Tue, Jun 19, 2012 at 10:09 AM, Neha Narkhede <neha.narkh...@gmail.com> >>> wrote: >>> > One of the goals of thinning the Kafka consumer client is removing the >>> > zookeeper client from the consumer. Without this, Kafka consumer >>> > client would depend on the stability of a zookeeper client. >>> >>> If there's a stability issue with the zookeeper client, then that >>> should be addressed. >>> >>> ZK is a fine tool for service discovery and coordination. It seems >>> like any new system that forced me, as a consumer, to use yet another >>> system to bootstrap and discover where my brokers are for a topic >>> would be a step backward. >>> >>> I'm curious, why, specifically, is removing ZK a design goal >>> (especially when it's such a core component of the broker)? I think >>> of other projects, like HBase, which seem to have no issue with using >>> ZK in their client. >>> >>> --Dave >>>
