Using storm facility for that purpose seems ill suited; for a starter, by the time you successfully suspended the spout, you probably would have violated the rate limit a bit etc.
You should be able to do operations that use one token within one thread, so you could simply backoff on rate limit notification, or use some in-memory rate limiter like Guava's RateLimiter if for some reason you need to use more than one thread. On Fri, Feb 21, 2014 at 8:38 AM, Danijel Schiavuzzi <[email protected]>wrote: > Hi Phil, > > I believe you can call Nimbus via it's Thrift interface and call the > "Deactivate" and "Activate" functions, which pause and resume the topology > spouts, respectively. > > But in your case, I believe a better architecture would be to store the > data into an intermediate queue, like Kafka. That way, you could scale your > topology (Storm cluster) as needed to support the required throughput. > > Best regards, > > Danijel > > > On Fri, Feb 21, 2014 at 4:33 AM, Phil Burress <[email protected]>wrote: > >> Is there a way to suspend a spout for a pre-determined period of time >> from within that spout or an attached bolt? >> As an example, assume that in your Topology you have a bolt retrieving >> data from Twitter's api, but you get rate limited. To me the obvious >> solution is to somehow suspend the spout for a period of time and then >> resume processing. Is that possible? >> >> If that cannot be accomplished with the current architecture, are there >> any other facilities in storm to handle a problem like this? >> >> Thanks very much! >> > > > > -- > Danijel Schiavuzzi >
