Jeff, This is definitely a strong use case for nifi.
It might be that InvokeHTTP is the better choice here. If what you'd like to do is effectively throttle the rate at which you hit the web service with the InvokeHttp calls you can schedule that processor to run as often as you like (for example every 100 ms). Then use backpressure settings on the queues feeding that InvokeHTTP process. Effectively you can control where data will back up in the flow while it is being throttled. If the lookup data is a good candidate for caching then there may be other great options to make this more efficient. Perhaps you can share a flow template of what you have so far and we can make recommendations on next steps? Thanks Joe On Fri, Feb 12, 2016 at 12:28 AM, Jeff - Data Bean Australia <[email protected]> wrote: > Hi > > I got a use case like this: > > There is a file that contains thousands of items, each on one line. For each > item, it will trigger one GetHTTP processor to fetch some data. > > Here is what I am trying to do: > > 1. Fetch this file > 2. For each line, I generate one file using SplitText > 3. Drive GetHTTP downstream. > > However, given there are more than 2000 lines, more than 2000 HTTP Get > processes will be created and flooded into one web site, which doesn't sound > like a good idea. So I would like to control the processors, so that only a > couple of them will be running at the same time, and maybe delay for a > couple of seconds after finish. > > How can I do that in NiFi? > > Thanks, > Jeff > > > > > -- > Data Bean - A Big Data Solution Provider in Australia.
