Russ, This sort of deviates from your original question, but Would applying a flowfile expiration time on the connection (during experimentation) work with your flow? This would keep the queue more manageable.
> On Jan 10, 2017, at 4:35 PM, Russell Bateman <[email protected]> wrote: > > To update this thread, ... > > 1. Setting up a no-op processor to "drain" the queue doesn't seem to present > any speed advantage over right-clicking the queue and choosing Empty queue. > 2. Removing the flowfile and provenance repositories (cd flowfile_repository > ; rm -rf *) is instantaneous. > 3. However, removing the content repository from the filesystem via console > isn't immediate. It does take time. It appears that it may not be taking as > long as either method in #1 above, but it does take a very long time. I had > over a hundred million files being emptied when I started this thread and I'm > still only down just under 40% left as I write this final volley 2 hours > after trying to delete them using the filesystem (CentOS 7, CPU inactive, > 128Gb memory, 56 cores, hard drive--not SSD). > 4. I don't dare delete the database_repository. > 5. I'm assuming that once all three repositories are gone, I'll be able to > restart NiFi without any damage to what I expect flowfile.xml.gz and the rest > of the conf/ subdirectory are safe-guarding for me. > > There may not be any instantaneous solution anyone can offer short of > renaming the content repository subdirectory, setting up a background task to > smoke it, and creating a new content repository to start afresh with. I > haven't tried that yet. > > It's likely a better idea to think ahead about this and provide for draining > the queues as the flowfiles reach them. If all you want is a count of > successful outcomes, you could do that with a no-op processor that counts as > it goes and puts the number somewhere for safe-keeping. I wouldn't be doing > this if I weren't trying to make some observations on performance, processing > loads, etc., in short, testing. > > If I experience anything nasty or noteworthy from this point on [4,5], I'll > come back and update this thread again. > > >> On 01/10/2017 02:50 PM, Russell Bateman wrote: >> In my case, I'm experimenting with huge flows and huge numbers of files. I >> wasn't thinking about how much work I'd create for myself by storing up >> files in a queue at the end (or, in some cases, at intermediate points) when >> I might want to clean house and start over. >> >> So, I can just bring NiFi down, smoke the repos, then restart safely? >> >> >>> On 01/10/2017 02:39 PM, Joe Witt wrote: >>> Millions or gajillions will indeed take a while as they have to swap in as >>> presently implemented. We could certainly optimize that if is a common >>> need. >>> >>> Blowing away the repos will certainly do the trick and be faster. Though >>> is clearly a blunt instrument. >>> >>> Do you think we need an express queue killer option? >>> >>> On Jan 10, 2017 1:32 PM, "Russell Bateman" <[email protected]> wrote: >>> If I'm experimenting and have gajillions of flowfiles in a queue that takes >>> a very long time to empty from the UI, is there a quicker way? I can >>> certainly bounce NiFi, delete files, both, etc. >>> >> >
