thanks guys, it's clear now. - inder
On Mon, Oct 17, 2011 at 9:51 PM, Neha Narkhede <neha.narkh...@gmail.com>wrote: > Inder, > > From the producer side,VIP and the ZK Producer are mutually exclusive > options. > You use either one or the other. > > > 1. When a new broker comes in in the cluster, it would register in broker > > registry but won't be subscribed to any topics. > > If you are using a VIP, then it will route some of the new requests to the > new > broker automatically. If you are using the ZK Producer, it will get > notified once the > new broker registers itself in ZK, and it will start routing new > requests to the broker > according to the partitioner you use (Default is random partitioning, > which is similar to > VIP behavior) > > > 3. If we are using a VIP to front brokers, how do they get all the > > information from ZK about what topics/partitions should go to which > broker > > for appropriate routing. > > As I said, if you are using VIP, then brokers are not "discovered" > through ZK watchers. > The VIP should have a TCP healthcheck on the Kafka port, and all > brokers that have > Kafka running on that port, will be involved in receiving Kafka data. > The VIP randomly > distributes produce requests amongst the live brokers. > > Thanks, > Neha > > > On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.p...@gmail.com> wrote: > > Jun, > > > > appreciate your prompt response. > > > > need some more inputs in understanding this better. > > Is the following understanding appropriate. > > > > 1. When a new broker comes in in the cluster, it would register in broker > > registry but won't be subscribed to any topics. > > 2. re-balance triggered by consumers will get some partitions of existing > > topics to this broker. > > 3. If we are using a VIP to front brokers, how do they get all the > > information from ZK about what topics/partitions should go to which > broker > > for appropriate routing. > > > > Thanks, > > - Inder > > Yahoo India R&D. > > > > On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <jun...@gmail.com> wrote: > > > >> Inder, > >> > >> 1. Existing data doesn't get moved to the new broker. Some of the new > data > >> will get routed to the new broker once the producer recognizes the new > >> broker (either through ZK-based producer or vip). > >> > >> 2. Yes. Once the new broker gets data for a new topic, consumers will be > >> notified and start rebalance. > >> > >> Thanks, > >> > >> Jun > >> > >> On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.p...@gmail.com> > wrote: > >> > >> > Hi, > >> > > >> > I am trying to understand kafka by going through documents/wiki's for > >> > usage. > >> > Please pardon if the dev mailing list isn't appropriate to be in the > TO > >> > for > >> > this query. > >> > > >> > Couple of queries. > >> > > >> > 1. If you add new broker boxes, how will existing topics and their > >> > partitions be redistributed on the new broker. > >> > 2. Does [1] trigger a re-balance process via watchers in which case if > >> you > >> > wanna redistribute only certain event types on the new broker, can it > be > >> > achieved dynamically. > >> > > >> > > >> > -- Inder > >> > > >> > > > > > > > > -- > > -- Inder > > > -- -- Inder
