Yes, I had thought of that, but I needed to know how many reached the
end and I lamely thought I could look at the queue size not thinking
about how emptying the queue wouldn't be instantaneous.
I finally just put a counting processor we wrote (that maintains named
NiFi counters) in each of the flows I'm testing, then let the flowfiles
drain out into the bit bucket.
This question really only arose because I was concentrating on my
testing and not the bigger picture of what hundreds of millions of
flowfiles might do to me in the end. So, I guess this is sort of a
wasted thread except that (I'm hoping) it will be Googlable by someone
else down the road who wonders as I did.
Thanks
On 01/11/2017 12:08 AM, Lee Laim wrote:
Russ,
This sort of deviates from your original question, but Would applying
a flowfile expiration time on the connection (during experimentation)
work with your flow? This would keep the queue more manageable.
On Jan 10, 2017, at 4:35 PM, Russell Bateman <[email protected]
<mailto:[email protected]>> wrote:
To update this thread, ...
1. Setting up a no-op processor to "drain" the queue doesn't seem to
present any speed advantage over right-clicking the queue and
choosing Empty queue.
2. Removing the flowfile and provenance repositories (cd
flowfile_repository ; rm -rf *) is instantaneous.
3. However, removing the content repository from the filesystem via
console isn't immediate. It does take time. It appears that it may
not be taking as long as either method in #1 above, but it does take
a very long time. I had over a hundred million files being emptied
when I started this thread and I'm still only down just under 40%
left as I write this final volley 2 hours after trying to delete them
using the filesystem (CentOS 7, CPU inactive, 128Gb memory, 56 cores,
hard drive--not SSD).
4. I don't dare delete the /database_repository/.
5. I'm assuming that once all three repositories are gone, I'll be
able to restart NiFi without any damage to what I expect
/flowfile.xml.gz/ and the rest of the /conf// subdirectory are
safe-guarding for me.
There may not be any instantaneous solution anyone can offer short of
renaming the content repository subdirectory, setting up a background
task to smoke it, and creating a new content repository to start
afresh with. I haven't tried that yet.
It's likely a better idea to think ahead about this and provide for
draining the queues as the flowfiles reach them. If all you want is a
count of successful outcomes, you could do that with a no-op
processor that counts as it goes and puts the number somewhere for
safe-keeping. I wouldn't be doing this if I weren't trying to make
some observations on performance, processing loads, etc., in short,
testing.
If I experience anything nasty or noteworthy from this point on
[4,5], I'll come back and update this thread again.
On 01/10/2017 02:50 PM, Russell Bateman wrote:
In my case, I'm experimenting with huge flows and huge numbers of
files. I wasn't thinking about how much work I'd create for myself
by storing up files in a queue at the end (or, in some cases, at
intermediate points) when I might want to clean house and start over.
So, I can just bring NiFi down, smoke the repos, then restart safely?
On 01/10/2017 02:39 PM, Joe Witt wrote:
Millions or gajillions will indeed take a while as they have to
swap in as presently implemented. We could certainly optimize that
if is a common need.
Blowing away the repos will certainly do the trick and be faster.
Though is clearly a blunt instrument.
Do you think we need an express queue killer option?
On Jan 10, 2017 1:32 PM, "Russell Bateman" <[email protected]
<mailto:[email protected]>> wrote:
If I'm experimenting and have gajillions of flowfiles in a
queue that takes a very long time to empty from the UI, is
there a quicker way? I can certainly bounce NiFi, delete files,
both, etc.
