Thanks for the quick response! Unfortunately, we're forced to use HTTP. Any ideas?
On Fri, Sep 26, 2014 at 5:07 PM, Supun Kamburugamuva <supu...@gmail.com> wrote: > On Fri, Sep 26, 2014 at 10:49 AM, Peter Neumark <peter.neum...@prezi.com> > wrote: > >> Hi all, >> >> We want to replace a legacy custom app with storm, but -being storm >> newbies- we're not sure what's the best way to solve the following problem: >> >> An HTTP endpoint returns the list of events which occurred between two >> timestamps. The task is to continuously poll this event source for new >> events, optionally perform some transformation and aggregation operations >> on them, and finally make an HTTP request to an endpoint with some events. >> >> We thought of a simple topology: >> 1. A clock-spout determines which time interval to process. >> 2. A bolt takes the time interval as input, and fetches the event list >> for that interval fro the event source, emitting them as individual tuples. >> 3. After some processing of the tuples, we aggregate them into fixed size >> groups, which we send in HTTP requests to an event sink. >> >> The big question is how to make sure that all events are successfully >> delivered to the event sink. I know storm guarantees the delivery of tuples >> within the topology, but how could I guarantee that the HTTP requests to >> the event sink are also successful (and retried if necessary). >> > > I think this is not a question about Storm and rather a question about how > to deliver a message reliably to some sink. From my experience it is bit > hard to achieve something like this with HTTP. This functionality is built > in to message brokers like RabbitMQ, ActiveMQ, Kafka etc and if you use a > broker to send your events to the sink you can get a delivery guarantee. > > Thanks, > Supun.. > > >> >> All help, suggestions and pointers welcome! >> Peter >> >> -- >> >> *Peter Neumark* >> DevOps guy @Prezi <http://prezi.com> >> > > > > -- > Supun Kamburugamuva > Member, Apache Software Foundation; http://www.apache.org > E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 > Blog: http://supunk.blogspot.com > > -- *Peter Neumark* DevOps guy @Prezi <http://prezi.com>
