may be this is useful https://github.com/ptgoetz/storm-signals
On Fri, Feb 21, 2014 at 10:02 AM, Enno Shioji <[email protected]> wrote: > Using storm facility for that purpose seems ill suited; for a starter, by > the time you successfully suspended the spout, you probably would have > violated the rate limit a bit etc. > > You should be able to do operations that use one token within one thread, > so you could simply backoff on rate limit notification, or use some > in-memory rate limiter like Guava's RateLimiter if for some reason you need > to use more than one thread. > > > > On Fri, Feb 21, 2014 at 8:38 AM, Danijel Schiavuzzi <[email protected]>wrote: > >> Hi Phil, >> >> I believe you can call Nimbus via it's Thrift interface and call the >> "Deactivate" and "Activate" functions, which pause and resume the topology >> spouts, respectively. >> >> But in your case, I believe a better architecture would be to store the >> data into an intermediate queue, like Kafka. That way, you could scale your >> topology (Storm cluster) as needed to support the required throughput. >> >> Best regards, >> >> Danijel >> >> >> On Fri, Feb 21, 2014 at 4:33 AM, Phil Burress >> <[email protected]>wrote: >> >>> Is there a way to suspend a spout for a pre-determined period of time >>> from within that spout or an attached bolt? >>> As an example, assume that in your Topology you have a bolt retrieving >>> data from Twitter's api, but you get rate limited. To me the obvious >>> solution is to somehow suspend the spout for a period of time and then >>> resume processing. Is that possible? >>> >>> If that cannot be accomplished with the current architecture, are there >>> any other facilities in storm to handle a problem like this? >>> >>> Thanks very much! >>> >> >> >> >> -- >> Danijel Schiavuzzi >> > > -- With Regards, Vinoth Kumar K
