Inder, If you just want to do random distribution, ZK-based load balancing is not too much different from VIP. We have experienced some subtle issues with VIP. For example, it's not easy to figure out the real IP of a client connected to a broker. Also, VIP timeout can be tricky (need to send RST to the client if connection is killed by timeouts).
Further more, if you want to partition based on a key, then this is hard for VIP to do. Thanks, Jun On Sun, Dec 11, 2011 at 9:20 PM, Inder Pall <[email protected]> wrote: > Neha, > > how would it be better than putting the new broker behind a VIP. > Is the ZK based producer equipped with more info to take a better call. > > > > On Sun, Dec 11, 2011 at 11:57 PM, Neha Narkhede <[email protected] > >wrote: > > > Inder, > > > > Zookeeper is used on the producer side for load balancing. It is used for > > automatic detection of new and failed brokers. > > > > Thanks, > > Neha > > > > On Sun, Dec 11, 2011 at 2:44 AM, Inder Pall <[email protected]> > wrote: > > > > > why do u want to integrate zk at producer side. can you please > elaborate > > > your usecase. > > > > > > is it for load balancing. > > > On Dec 7, 2011 12:35 AM, "Jun Rao" <[email protected]> wrote: > > > > > > > > David, > > > > > > > > Could you also comment whether you want a programming api or a > > monitoring > > > > api? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Tue, Dec 6, 2011 at 10:03 AM, David Ormsbee <[email protected]> > > > wrote: > > > > > > > > > Jun, Neha: > > > > > > > > > > Thank you for pointing me the right way on the Producer issue. :-) > > > > > > > > > > As for the metadata requests, I've filed a ticket here: > > > > > > > > > > https://issues.apache.org/jira/browse/KAFKA-223 > > > > > > > > > > Take care. > > > > > > > > > > Dave > > > > > > > > > > On Tue, Dec 6, 2011 at 12:05 PM, Neha Narkhede < > > > [email protected] > > > > > >wrote: > > > > > > > > > > > Hi Dave, > > > > > > > > > > > > >> Or should I just make the initial assumption > > > > > > that all brokers have one partition, send messages to those to > > create > > > the > > > > > > topic on the brokers, and then readjust when the the brokers > > publish > > > > > their > > > > > > partitions-per-topic information to ZK after the initial writes? > > > > > > > > > > > > That's exactly what we do. You can take a look at > > > > > > bootstrapWithExistingBrokers() in > > > kafka.producer.ZKBrokerPartitionInfo. > > > > > > > > > > > > >> As an aside, is there any way to query metadata about the > broker > > > > > > directly? > > > > > > So that a simple consumer might be able to ask a broker for what > > > topics > > > > > > they have, how many partitions, what version it's running, etc? > > > > > > > > > > > > Today, we don't have such APIs. But I was thinking we can have > > > something > > > > > > similar to the 4-letter commands in zookeeper. > > > > > > For example, > > > > > > > > > > > > echo topics | nc broker-host broker-port (Returns all topics > > > with > > > > > > number of partitions on each broker) > > > > > > echo srvr | nc broker-host broker-port (Returns basic > > broker > > > > > stats > > > > > > like version, number of topics etc) > > > > > > > > > > > > If the above functionality is useful, lets file a JIRA for it. > > > > > > > > > > > > Thanks > > > > > > Neha > > > > > > > > > > > > On Tue, Dec 6, 2011 at 6:34 AM, David Ormsbee < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Hi folks, > > > > > > > > > > > > > > I'm implementing ZooKeeper support in our Python client, and I > > was > > > > > > > wondering what the expected behavior of the Producer is when > the > > > > > brokers > > > > > > > are completely empty. At this point, the brokers have > registered > > > > > > themselves > > > > > > > in /brokers/ids/[0..N], but there are no topics registered > under > > > > > > > /brokers/topics. If someone wants to create a Producer at this > > > point > > > > > and > > > > > > > start sending messages to a new topic, is that an error because > > no > > > > > > brokers > > > > > > > are registered for that topic? Or should I just make the > initial > > > > > > assumption > > > > > > > that all brokers have one partition, send messages to those to > > > create > > > > > the > > > > > > > topic on the brokers, and then readjust when the the brokers > > > publish > > > > > > their > > > > > > > partitions-per-topic information to ZK after the initial > writes? > > > > > > > > > > > > > > I'm sure I'm missing something obvious here, since we'd run > into > > a > > > > > > similar > > > > > > > problem for new, empty brokers joining the pool. > > > > > > > > > > > > > > As an aside, is there any way to query metadata about the > broker > > > > > > directly? > > > > > > > So that a simple consumer might be able to ask a broker for > what > > > topics > > > > > > > they have, how many partitions, what version it's running, etc? > > Or > > > is > > > > > > that > > > > > > > only ever published through ZooKeeper? > > > > > > > > > > > > > > Thank you. > > > > > > > > > > > > > > Dave > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > Thanks, > - Inder > Tech Platforms @Inmobi > Linkedin - http://goo.gl/eR4Ub >
