Murtaza, At LinkedIn, we are using the mirror maker tool in 0.7.1 for mirroring across DC.
Thanks, Jun On Tue, Jul 31, 2012 at 3:01 PM, Murtaza Doctor <murt...@richrelevance.com>wrote: > Hello Folks, > > We are trying to work out the Kafka deployment topology and need some > feedback from the community: > > Kafka Version: 0.7 (under test) and Zookeper to maintain offsets. > > At present, we have multiple live data centers (DC) just like LinkedIn and > a single backend data center. All the live colos are identical in terms of > the capacity and can serve the same functionality. We want to stream events > from each of these live DC's to the backend DC. I am assuming we have 3 > different options: > > 1. Have kafka brokers in each of the live DC's and then have kafka > brokers on the backend DC in a mirrored setup as described in this link: > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+mirroring which > Jay has put together. This probably mimics what LinkedIn has, please > correct me if I am wrong. > 2. Have a set of Kafka brokers in our backend DC and just have the > producers asynchronously send their messages to the broker. The option does > have lot of gaping holes specifically if the link goes down, In that event > we will have message loss specifically in the absence of message > replication which will be supported in 0.8 [Note we are going in production > with 0.7] > 3. Have brokers in each of the live data centers and then just have > consumers which reach from each of the brokers or in parallel from their > respective offsets. NO mirroring setup, but have the consumer be > responsible to read the messages. > > Would love to hear feedback on each of these deployment strategies and > their pros & cons. Additionally if there are other mechanisms to support > this which I have missed. > > Thanks, > murtaza >
