Jeff, Another approach I've used with some success is the ControlRate processor before the InvokeHttp, which gives you an intuitive way of limiting the number of requests in a specified time interval. Note however that this does need to be combined with back pressure control to prevent requests queuing behind the InvokeHttp. Feeding failure retries back into the ControlRate, or a funnel before the invoke also gives you a bit more control over groupings of retries for example.
Simon Sent from my iPhone - Simon Elliston Ball Solutions Engineer - EMEA +44 7930 424111<tel:+44%207930%20424111> Hortonworks - We Do Hadoop On 12 Feb 2016, at 05:34, Joe Witt <[email protected]<mailto:[email protected]>> wrote: Jeff, This is definitely a strong use case for nifi. It might be that InvokeHTTP is the better choice here. If what you'd like to do is effectively throttle the rate at which you hit the web service with the InvokeHttp calls you can schedule that processor to run as often as you like (for example every 100 ms). Then use backpressure settings on the queues feeding that InvokeHTTP process. Effectively you can control where data will back up in the flow while it is being throttled. If the lookup data is a good candidate for caching then there may be other great options to make this more efficient. Perhaps you can share a flow template of what you have so far and we can make recommendations on next steps? Thanks Joe On Fri, Feb 12, 2016 at 12:28 AM, Jeff - Data Bean Australia <[email protected]<mailto:[email protected]>> wrote: Hi I got a use case like this: There is a file that contains thousands of items, each on one line. For each item, it will trigger one GetHTTP processor to fetch some data. Here is what I am trying to do: 1. Fetch this file 2. For each line, I generate one file using SplitText 3. Drive GetHTTP downstream. However, given there are more than 2000 lines, more than 2000 HTTP Get processes will be created and flooded into one web site, which doesn't sound like a good idea. So I would like to control the processors, so that only a couple of them will be running at the same time, and maybe delay for a couple of seconds after finish. How can I do that in NiFi? Thanks, Jeff -- Data Bean - A Big Data Solution Provider in Australia.
