I get your reasoning at a high level. I should have specified that I wasn’t sure what Kafka does. I don’t have a hard software engineering background. I know that Kafka is “a message queuing” system, but I don’t really know what that means.
(I can’t believe you wrote all that from your iPhone....) B. From: Justin Workman Sent: Thursday, August 14, 2014 7:22 PM To: [email protected] Subject: Re: Kafka + Storm Personally, we looked at several options, including writing our own storm source. There are limited storm sources with community support out there. For us, it boiled down to the following; 1) community support and what appeared to be a standard method. Storm has now included the kafka source as a bundled component to storm. This made the implementation much faster, because the code was done. 2) the durability (replication and clustering) of Kafka. We have a three hour retention period on our queues, so if we need to do maintenance on storm or deploy an updated topology, we don't need to stop or replay any sources 3) the ability to have other tools attach to the Kafka queues to consume the same events for other purposes. 4) to compliment point #1, it's easy to write to Kafka. So it was little effort to start sending our desired data to Kafka. These are our main reasons ( I'm sure there were more ). Each use case is going to be different and Kafka might not be the best choice for everyone. For us it made sense. Justin Sent from my iPhone On Aug 14, 2014, at 6:08 PM, "Adaryl \"Bob\" Wakefield, MBA" <[email protected]> wrote: Can someone tell me why people put Kafka in front of Storm? Can’t Storm ingest messages without having Kafka in the middle? B.
