Re: Deleting millions of files from a queue...

*To:* users@nifi.apache.org *Subject:* Re: Deleting millions of files from a queue... Yes, I had thought of that, but I needed to know how many reached the end and I lamely thought I could look at the queue size not thinking about how emptying the queue wouldn't be instantaneous. I fina

RE: Deleting millions of files from a queue...

, but this is expensive to maintain. Regards, Kevin From: Russell Bateman [mailto:r...@windofkeltia.com] Sent: Wednesday, January 11, 2017 1:18 PM To: users@nifi.apache.org Subject: Re: Deleting millions of files from a queue... Yes, I had thought of that, but I needed to know how many reached

Re: Deleting millions of files from a queue...

Yes, I had thought of that, but I needed to know how many reached the end and I lamely thought I could look at the queue size not thinking about how emptying the queue wouldn't be instantaneous. I finally just put a counting processor we wrote (that maintains named NiFi counters) in each of

Re: Deleting millions of files from a queue...

Russ, This sort of deviates from your original question, but Would applying a flowfile expiration time on the connection (during experimentation) work with your flow? This would keep the queue more manageable. > On Jan 10, 2017, at 4:35 PM, Russell Bateman wrote: >

Re: Deleting millions of files from a queue...

To update this thread, ... 1. Setting up a no-op processor to "drain" the queue doesn't seem to present any speed advantage over right-clicking the queue and choosing Empty queue. 2. Removing the flowfile and provenance repositories (cd flowfile_repository ; rm -rf *) is instantaneous. 3.

Re: Deleting millions of files from a queue...

I'm trying your suggestion right now. It would certainly be an easy way to avoid accumulation, but, in terms of voiding a queue with millions of files saved up until I'm ready to lose them (at the end of a test run), it doesn't seem any improvement in speed over just right-clicking and

Re: Deleting millions of files from a queue...

In my case, I'm experimenting with huge flows and huge numbers of files. I wasn't thinking about how much work I'd create for myself by storing up files in a queue at the end (or, in some cases, at intermediate points) when I might want to clean house and start over. So, I can just bring NiFi

Re: Deleting millions of files from a queue...

If I want a sink node to get rid of flow files while I’m messing around I add a ‘dev/null' update attribute processor that does nothing. Set the output to automatically terminate and just connect the queue to it and start it up. If you want to retain some output to look at just stop it

Re: Deleting millions of files from a queue...

Millions or gajillions will indeed take a while as they have to swap in as presently implemented. We could certainly optimize that if is a common need. Blowing away the repos will certainly do the trick and be faster. Though is clearly a blunt instrument. Do you think we need an express queue

Deleting millions of files from a queue...

If I'm experimenting and have gajillions of flowfiles in a queue that takes a very long time to empty from the UI, is there a quicker way? I can certainly bounce NiFi, delete files, both, etc.