I do need more information, because I tried using that option, but the processor just continued filling the queue anyway, I told it to only allow 2000 before back pressure kicks in, but it kept going and I ended up with 5k files in the queue before I restarted Nifi to get the processor to stop.
On Wed, Jun 8, 2016 at 8:45 AM, Bryan Bende <[email protected]> wrote: > Hello, > > Take a look at the options available when right-clicking on a queue... > What you described is what NiFi calls back-pressure. You can configured a > queue to have an object threshold (# of flow files) or data size threshold > (total size of all flow files). > When one of these thresholds is reached, NiFi will no longer let the source > processor run until the condition goes back under the threshold. > > Let us know if you need any more info on this. > > Thanks, > > Bryan > > On Wed, Jun 8, 2016 at 10:40 AM, Shaine Berube < > [email protected]> wrote: > > > Hello all, > > > > I'm kind of new to developing Nifi, though I've been doing some pretty in > > depth stuff and some advanced database queries. My question is in > > regarding the queues between processor, I want to limit a queue to... say > > 2000, how would I go about doing that? Or better yet, how would I tell > the > > processor generating the queue to only put a max of 2000 files into the > > queue? > > > > Allow me to explain with a scenario: > > We are doing data migration from one database to another. > > -Processor A is generating a queue consumed by Processor B > > -Processor A is taking configuration and generating SQL queries in 1000 > > record chunks so that Processor B can insert them into a new database. > > Given the size of the source database, Processor A can potentially > generate > > hundreds of thousands of files. > > > > Is there a way for Processor A to check it's down stream queue for the > > queue size? How would I get Processor A to only put 2000 files into the > > queue at any given time, so that Processor A can continue running but > wait > > for room in the queue? > > > > Thank you in advance. > > > > -- > > *Shaine Berube* > > > -- *Shaine Berube*
