Your approach looks interesting and I personally never thought of graceful shutdown of topology. Here just one small point that sometimes you may get callback to fail method also. Callback may not be to ack method always. Overall your approach looks little complicate but looks good and guarantees the expected results.
On Mon, Dec 22, 2014 at 9:19 PM, Vincent Rischmann <[email protected]> wrote: > Hi, > > I'm trying to find a reliable way to gracefully shutdown a Trident > topology, meaning when we want to stop it, it processes all remaining > tuples but the spout does not emit anymore tuples. > > Right now I have done something like this: > > - when we want to stop it, a flag is set > > - keep track of pending tasks with a list of batchId > > - each call to emitBatch in the spout will check if the topology is > "paused" > > - if paused, do nothing > > - if not paused, emit a tuple by taking data from a queue > > - each call to ack(batchId) will remove the corresponding batchId from the > pending tasks > > - if no more pending tasks, and the stop flag is set, notify by sending a > event via Redis > > > First, is this a correct approach in your opinion ? Is it guaranteed that > ack() will be called for each batchId ? > > Second, I'm curious to hear about how everyone does graceful stopping. It > seems to me it's something a lot of people would want yet I didn't find a > lot of information about it. > > Vincent. >
