> What do you see as the downside to running a KAFKA Broker and letting > it write your files locally? >
Here is every downside I can come up with. Some are trivial, but I've tried to play devil's advocate. 1) Additional memory/processing footprint, as you mentioned. 2) Additional network usage by Kafka consumers hitting producer box. 3) Loss of either producer elasticity or message retention. One of the awesome features of Kafka is being able to hold on a long history of messages and replay as needed. But if the producers machines hold my brokers, I can no longer scale down the number of producer machines as the system load drops--a loss of elasticity. This ties to another group discussion about decommissioning brokers. 4) Security... some producers live on web-accessible machines. The less open ports accepting incoming connections, the better. Not a huge issue with good firewall rules, but still something to ponder. 5) Loss of data duplication has downsides too. Having multiple copies of data does violate DRY, but it can also add robustness. If the hard-disk on either the producer or the brokers dies, you still have the data lying around. Since the data is not getting changed on either the broker or any producer log files, you shouldn't have syncing issues normally associated with DRY violations. - Niek
