Inder, >From the producer side,VIP and the ZK Producer are mutually exclusive options. You use either one or the other.
> 1. When a new broker comes in in the cluster, it would register in broker > registry but won't be subscribed to any topics. If you are using a VIP, then it will route some of the new requests to the new broker automatically. If you are using the ZK Producer, it will get notified once the new broker registers itself in ZK, and it will start routing new requests to the broker according to the partitioner you use (Default is random partitioning, which is similar to VIP behavior) > 3. If we are using a VIP to front brokers, how do they get all the > information from ZK about what topics/partitions should go to which broker > for appropriate routing. As I said, if you are using VIP, then brokers are not "discovered" through ZK watchers. The VIP should have a TCP healthcheck on the Kafka port, and all brokers that have Kafka running on that port, will be involved in receiving Kafka data. The VIP randomly distributes produce requests amongst the live brokers. Thanks, Neha On Mon, Oct 17, 2011 at 9:01 AM, Inder Pall <inder.p...@gmail.com> wrote: > Jun, > > appreciate your prompt response. > > need some more inputs in understanding this better. > Is the following understanding appropriate. > > 1. When a new broker comes in in the cluster, it would register in broker > registry but won't be subscribed to any topics. > 2. re-balance triggered by consumers will get some partitions of existing > topics to this broker. > 3. If we are using a VIP to front brokers, how do they get all the > information from ZK about what topics/partitions should go to which broker > for appropriate routing. > > Thanks, > - Inder > Yahoo India R&D. > > On Mon, Oct 17, 2011 at 8:21 PM, Jun Rao <jun...@gmail.com> wrote: > >> Inder, >> >> 1. Existing data doesn't get moved to the new broker. Some of the new data >> will get routed to the new broker once the producer recognizes the new >> broker (either through ZK-based producer or vip). >> >> 2. Yes. Once the new broker gets data for a new topic, consumers will be >> notified and start rebalance. >> >> Thanks, >> >> Jun >> >> On Sun, Oct 16, 2011 at 10:29 PM, Inder Pall <inder.p...@gmail.com> wrote: >> >> > Hi, >> > >> > I am trying to understand kafka by going through documents/wiki's for >> > usage. >> > Please pardon if the dev mailing list isn't appropriate to be in the TO >> > for >> > this query. >> > >> > Couple of queries. >> > >> > 1. If you add new broker boxes, how will existing topics and their >> > partitions be redistributed on the new broker. >> > 2. Does [1] trigger a re-balance process via watchers in which case if >> you >> > wanna redistribute only certain event types on the new broker, can it be >> > achieved dynamically. >> > >> > >> > -- Inder >> > >> > > > > -- > -- Inder >
