Thanks very much for the replies; all great information and suggestions. One last quick question. Outside of tuples, is there any other mechanism for bolts and spouts to communicate with each other out-of-the-box?
On Fri, Feb 21, 2014 at 4:44 AM, Vinoth Kumar Kannan <[email protected]>wrote: > may be this is useful > > https://github.com/ptgoetz/storm-signals > > > On Fri, Feb 21, 2014 at 10:02 AM, Enno Shioji <[email protected]> wrote: > >> Using storm facility for that purpose seems ill suited; for a starter, by >> the time you successfully suspended the spout, you probably would have >> violated the rate limit a bit etc. >> >> You should be able to do operations that use one token within one thread, >> so you could simply backoff on rate limit notification, or use some >> in-memory rate limiter like Guava's RateLimiter if for some reason you need >> to use more than one thread. >> >> >> >> On Fri, Feb 21, 2014 at 8:38 AM, Danijel Schiavuzzi >> <[email protected]>wrote: >> >>> Hi Phil, >>> >>> I believe you can call Nimbus via it's Thrift interface and call the >>> "Deactivate" and "Activate" functions, which pause and resume the topology >>> spouts, respectively. >>> >>> But in your case, I believe a better architecture would be to store the >>> data into an intermediate queue, like Kafka. That way, you could scale your >>> topology (Storm cluster) as needed to support the required throughput. >>> >>> Best regards, >>> >>> Danijel >>> >>> >>> On Fri, Feb 21, 2014 at 4:33 AM, Phil Burress >>> <[email protected]>wrote: >>> >>>> Is there a way to suspend a spout for a pre-determined period of time >>>> from within that spout or an attached bolt? >>>> As an example, assume that in your Topology you have a bolt retrieving >>>> data from Twitter's api, but you get rate limited. To me the obvious >>>> solution is to somehow suspend the spout for a period of time and then >>>> resume processing. Is that possible? >>>> >>>> If that cannot be accomplished with the current architecture, are there >>>> any other facilities in storm to handle a problem like this? >>>> >>>> Thanks very much! >>>> >>> >>> >>> >>> -- >>> Danijel Schiavuzzi >>> >> >> > > > -- > With Regards, > Vinoth Kumar K >
