>> how to setup kafka server (the bus) so that I can have > 1 server with redundancies for my producers and consumers and how it all works.
We have a mirroring feature where you can setup 2 clusters, one to be a mirror of the other. At LinkedIn, we have a production Kafka cluster, and an analytics Kafka cluster that mirrors the production one in real time. We still haven't updated the documentation to describe this in detail. Until replication is out, does the above mirroring approach work for you ? >> I am also hoping to hook up an open source consumer for Cassandra as we will be using Kafka day one for aggregating metrics to Cassandra (which we have already started open sourcing which is all Scala based). Exciting ! Let us know how that goes for you. A question I have - how do you think you will handle the "at least" delivery guarantees of Kafka here ? Thanks, Neha On Fri, Nov 4, 2011 at 8:56 AM, Jun Rao <jun...@gmail.com> wrote: > Joe, > > Setting up multiple brokers on different machines is pretty easy. You just > need to give each broker a unique broker id and give them the same ZK > connect string. By default, producers will spread the data randomly among > all brokers. There is currently no redundancy across brokers. So if a > broker is permanently gone, all unconsumed data on it is lost. We will be > working on replication (KAFKA-50) in the next release. > > Thanks, > > Jun > > On Fri, Nov 4, 2011 at 8:46 AM, Joe Stein <crypt...@gmail.com> wrote: > > > Hey folks, I have been digging on and off the last couple of days to > > figure/find out how to setup kafka server (the bus) so that I can have > > 1 > > server with redundancies for my producers and consumers and how it all > > works. I could have easily missed it with learning everything else about > > the system. I guess in general this is something that I still have not > > grocked and figured before I started to dig through the code for the > first > > time reach out to you folks to see if there was any information already > put > > together or tips/info I could find out or even just get pointed into the > > code specifically where messages are replicated and stored. I also had a > > bunch of questions in this regard to configurations for hardware best > > practices based on usage but figure once I get it up and running I will > > just run load tests and come up with my own needs for RAM, CPU and DISK > > (and JVM configurations as such) based on # message and data for each > > message. > > > > Everything is very awesome so far and I think this is going to work > > extremely well for a lot of what we do in production with a lot less > copies > > and no longer sequentially. I am also hoping to hook up an open source > > consumer for Cassandra as we will be using Kafka day one for aggregating > > metrics to Cassandra (which we have already started open sourcing which > is > > all Scala based). > > > > Many thanx! > > > > /* > > Joe Stein > > http://www.linkedin.com/in/charmalloc > > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > > */ > > >
