If we don't care about how many times the message is delivered (at least once) then we can use some error handling in HTTP to achieve a guarantee. You can use a request/response in HTTP and until you get a HTTP 200/202 you can retry the delivery. To get a exactly once guarantees we may need to go through some more complicated protocol.
On Fri, Sep 26, 2014 at 11:10 AM, Peter Neumark <peter.neum...@prezi.com> wrote: > Thanks for the quick response! > Unfortunately, we're forced to use HTTP. > Any ideas? > > On Fri, Sep 26, 2014 at 5:07 PM, Supun Kamburugamuva <supu...@gmail.com> > wrote: > >> On Fri, Sep 26, 2014 at 10:49 AM, Peter Neumark <peter.neum...@prezi.com> >> wrote: >> >>> Hi all, >>> >>> We want to replace a legacy custom app with storm, but -being storm >>> newbies- we're not sure what's the best way to solve the following problem: >>> >>> An HTTP endpoint returns the list of events which occurred between two >>> timestamps. The task is to continuously poll this event source for new >>> events, optionally perform some transformation and aggregation operations >>> on them, and finally make an HTTP request to an endpoint with some events. >>> >>> We thought of a simple topology: >>> 1. A clock-spout determines which time interval to process. >>> 2. A bolt takes the time interval as input, and fetches the event list >>> for that interval fro the event source, emitting them as individual tuples. >>> 3. After some processing of the tuples, we aggregate them into fixed >>> size groups, which we send in HTTP requests to an event sink. >>> >>> The big question is how to make sure that all events are successfully >>> delivered to the event sink. I know storm guarantees the delivery of tuples >>> within the topology, but how could I guarantee that the HTTP requests to >>> the event sink are also successful (and retried if necessary). >>> >> >> I think this is not a question about Storm and rather a question about >> how to deliver a message reliably to some sink. From my experience it is >> bit hard to achieve something like this with HTTP. This functionality is >> built in to message brokers like RabbitMQ, ActiveMQ, Kafka etc and if you >> use a broker to send your events to the sink you can get a delivery >> guarantee. >> >> Thanks, >> Supun.. >> >> >>> >>> All help, suggestions and pointers welcome! >>> Peter >>> >>> -- >>> >>> *Peter Neumark* >>> DevOps guy @Prezi <http://prezi.com> >>> >> >> >> >> -- >> Supun Kamburugamuva >> Member, Apache Software Foundation; http://www.apache.org >> E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 >> Blog: http://supunk.blogspot.com >> >> > > > -- > > *Peter Neumark* > DevOps guy @Prezi <http://prezi.com> > -- Supun Kamburugamuva Member, Apache Software Foundation; http://www.apache.org E-mail: supu...@gmail.com; Mobile: +1 812 369 6762 Blog: http://supunk.blogspot.com
