Hi all, We want to replace a legacy custom app with storm, but -being storm newbies- we're not sure what's the best way to solve the following problem:
An HTTP endpoint returns the list of events which occurred between two timestamps. The task is to continuously poll this event source for new events, optionally perform some transformation and aggregation operations on them, and finally make an HTTP request to an endpoint with some events. We thought of a simple topology: 1. A clock-spout determines which time interval to process. 2. A bolt takes the time interval as input, and fetches the event list for that interval fro the event source, emitting them as individual tuples. 3. After some processing of the tuples, we aggregate them into fixed size groups, which we send in HTTP requests to an event sink. The big question is how to make sure that all events are successfully delivered to the event sink. I know storm guarantees the delivery of tuples within the topology, but how could I guarantee that the HTTP requests to the event sink are also successful (and retried if necessary). All help, suggestions and pointers welcome! Peter -- *Peter Neumark* DevOps guy @Prezi <http://prezi.com>
