Niek, Thanks for the response. I agree with your assessments.
It's a beautiful thing that I don't have to commit to this decision early in the build process. I can start it one way and just switch to the other, as it is all a matter of configuration and not code. I'll let you know if we learn anything interesting along the way. Ed On Thu, Apr 12, 2012 at 5:38 PM, Niek Sanders <niek.sand...@gmail.com> wrote: >> What do you see as the downside to running a KAFKA Broker and letting >> it write your files locally? >> > > Here is every downside I can come up with. Some are trivial, but I've > tried to play devil's advocate. > > 1) Additional memory/processing footprint, as you mentioned. > > 2) Additional network usage by Kafka consumers hitting producer box. > > 3) Loss of either producer elasticity or message retention. One of > the awesome features of Kafka is being able to hold on a long history > of messages and replay as needed. But if the producers machines hold > my brokers, I can no longer scale down the number of producer machines > as the system load drops--a loss of elasticity. This ties to another > group discussion about decommissioning brokers. > > 4) Security... some producers live on web-accessible machines. The > less open ports accepting incoming connections, the better. Not a > huge issue with good firewall rules, but still something to ponder. > > 5) Loss of data duplication has downsides too. Having multiple copies > of data does violate DRY, but it can also add robustness. If the > hard-disk on either the producer or the brokers dies, you still have > the data lying around. Since the data is not getting changed on > either the broker or any producer log files, you shouldn't have > syncing issues normally associated with DRY violations. > > > - Niek
