Thanks Mike. What if I increase the parallelism of the long running bolt, so that other threads would still go thru it when export is not required? Will ONE bolt thread taking a long time, still need me to increase the message timeout to a very high number?
I’ve been contemplating sending the long running bolt’s task to a German job queue asynchronously and have it process there instead of Storm. -- Subrat Basnet Sent with Sparrow (http://www.sparrowmailapp.com/?sig) On Tuesday, June 2, 2015 at 6:29 PM, Mike Thomsen wrote: > Not that I know of. I don't think Storm has an effective mechanism for > allowing a particular tuple to take such an unusually long time while not > affecting the amount of time allotted to the other bolts and tuples. You > could jack those numbers up very, very high, but that could conceivably > encourage other bolts to take longer than you want instead of > failing/replaying a tuple. > > If you REALLY want to use storm for this long processing of a single piece of > data, you could do something like this supposing it's just one bolt that does > the long processing: > > 1. Write the data to disk, hdfs, hbase, RDBMS, etc. > 2. Write a new topology based on a signal spout (zookeeper signals) > 3. Give your new topology a ridiculously high amount of time for processing a > single tuple > 4. Have your current topology use SignalClient to post a zookeeper message > for the the new one, when the last tuple is ready to be processed > > On Tue, Jun 2, 2015 at 8:04 AM, Subrat Basnet <[email protected] > (mailto:[email protected])> wrote: > > Hi there, > > > > Is it normal to have long running bolts once in a while? When I say long > > running, I’m talking about a bolt that takes a few hours to process a tuple. > > > > I need to export data, push notifications and upload files with this when I > > reach the LAST tuple of a sequence of tuples. This does not happen on every > > tuple. > > > > I am worried, that this will make my whole topology hang up. My instinct is > > to give this particular bolt a higher parallelism, so that other threads > > are available to process when one bolt is hung up. > > > > Please advise, what would be the best way to achieve this. > > > > Thanks! > > Subrat > > > > -- > > Subrat Basnet > > Sent with Sparrow (http://www.sparrowmailapp.com/?sig) > > >
