Thanks Andrew. J
From: Andrew Montalenti [mailto:[email protected]] Sent: Thursday, June 05, 2014 3:30 PM To: [email protected] Subject: Re: Cluster Failure and Data Recovery Sounds like you might benefit from considering something like Kafka instead of a standard MQ. We have HYPERLINK "http://www.parsely.com/slides/logs/notes/#introducing-apache-kafka"some notes about this publicly online from our PyData talk on Kafka/Storm. You can configure Kafka to have an SLA on data that is in terms of data size or time; if your entire topology crashes or goes down, then you can resume messages at the spout from the moment the failure happened, and pay no penalty. (Of course, then you need to figure out how to guarantee your Kafka plant is always online, but this is do-able given its distributed architecture.) This doesn't sound like a problem that Storm should think about solving -- after all, if your entire Storm cluster fails, all of the high availability guarantees of each component are, by definition, out the window. On Thu, Jun 5, 2014 at 2:08 PM, Nima Movafaghrad <HYPERLINK "mailto:[email protected]"[email protected]> wrote: Thanks Srinath. We are already using the reliable message processing for bolts failure etc. My problem is with a catastrophic cases. For example, what happens if the entire cluster goes down or what if the Topology fully fails. At the moment we are reading from MQ and although keeping the transactions open would resolve our data loss prevention issue it isn't quiet feasible. Some of our bolts listen and batch for up to 30 seconds so they have big enough batches that can be committed to RDBMS. Keeping the transactions open for that long slows things down considerably. So I guess to frame to question better I should ask, if there a way to persist the intermediate data? Thanks, Nima From: Srinath C [mailto:HYPERLINK "mailto:[email protected]"[email protected]] Sent: Wednesday, June 04, 2014 5:49 PM To: user Subject: Re: Cluster Failure and Data Recovery Hi Nima, Use the HYPERLINK "https://github.com/nathanmarz/storm/wiki/Guaranteeing-message-processing"reliable message processing mechanism to ensure that there is no data loss. You would need support for transactional semantics from the tuple source where spout can commit/abort a read (kestrel, kafka, rabbitmq, etc can do this). Yes you would need to keep the queue transactions open until the spout receives an "ack" or "fail" for every tuple. IMO, this ensures that each tuple is processed "atleast once" and not "exactly once" so you need to be prepared to end up with duplicate entries in your DB or have a way to figure out that a write to DB is duplicate or earlier write. This is case where there are crashes with intermediate data in memory. Regards, Srinath. On Thu, Jun 5, 2014 at 5:37 AM, Nima Movafaghrad <HYPERLINK "mailto:[email protected]"[email protected]> wrote: Hi everyone, We are in the process of designing a high available system with zero data loss tolerance. Plan is for the spouts to read from a queue and process them down in several different specialized bolts and then flush to DB. How can we guarantee no data loss here? Should we keep the queue transactions open until data is committed to DB? Should we persist the state of all the bolts? What happens to the intermediate data if the whole cluster fails? Any suggestions are much appreciated. Nima
