Joe, Setting up multiple brokers on different machines is pretty easy. You just need to give each broker a unique broker id and give them the same ZK connect string. By default, producers will spread the data randomly among all brokers. There is currently no redundancy across brokers. So if a broker is permanently gone, all unconsumed data on it is lost. We will be working on replication (KAFKA-50) in the next release.
Thanks, Jun On Fri, Nov 4, 2011 at 8:46 AM, Joe Stein <crypt...@gmail.com> wrote: > Hey folks, I have been digging on and off the last couple of days to > figure/find out how to setup kafka server (the bus) so that I can have > 1 > server with redundancies for my producers and consumers and how it all > works. I could have easily missed it with learning everything else about > the system. I guess in general this is something that I still have not > grocked and figured before I started to dig through the code for the first > time reach out to you folks to see if there was any information already put > together or tips/info I could find out or even just get pointed into the > code specifically where messages are replicated and stored. I also had a > bunch of questions in this regard to configurations for hardware best > practices based on usage but figure once I get it up and running I will > just run load tests and come up with my own needs for RAM, CPU and DISK > (and JVM configurations as such) based on # message and data for each > message. > > Everything is very awesome so far and I think this is going to work > extremely well for a lot of what we do in production with a lot less copies > and no longer sequentially. I am also hoping to hook up an open source > consumer for Cassandra as we will be using Kafka day one for aggregating > metrics to Cassandra (which we have already started open sourcing which is > all Scala based). > > Many thanx! > > /* > Joe Stein > http://www.linkedin.com/in/charmalloc > Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop> > */ >
