Joe - I am interested in replication as well. We are going live with our implementation in the next few days.
In our setup events are fed into one of two datacenters (depending on which is closest to the client). We wanted to be able to process all of the data in a single DC regardless of which DC the event originally arrived in. So we wrote small project to make this possible. The project uses a fairly simple process that consumes data from one Kafka broker and then repeats the data to another Kafka broker (in our case the 'other' broker is in the remote data center). ZooKeeper is used to identify peers and detect failures, new nodes, etc. This isn't really replication but can be used repeat all data from one broker onto another and works for our use case. If there is general interest in this project we can work on open sourcing it. Matt On Thu, Dec 29, 2011 at 10:09 AM, Joe Stein <crypt...@gmail.com> wrote: > Hello, hope everyone's holiday break was/is going well? (If you are lucky > enough to have one). > > My busy season (mobile advertising) will start to ease up next week and I > will get to have some down time. > > We use Kafka in production now and are looking to up it's usage but really > need/want replication and outside of work it is an interesting problem to > work on. > > So, I wanted to follow-up on the replication work and what I can do to > help? I was not sure where this had left off and if the work had already > started or what. > > If not, is it good to have maybe have a conference call or something to go > through it and divy things up? If it has started let me know where I can > jump in please. > > Thanks!!! > > /* > Joe Stein > http://www.linkedin.com/in/charmalloc > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > */
