Depending on the size of the nodes, you may want to also increase the number of
indexing threads
("nifi.provenance.repository.index.threads"). You may want to also increase the
amount of time to store provenance
from 24 hours to something like 36 months... for most people just limiting
based on size is enough. The time is really
only useful if you are worried about it from a compliance standpoint, such that
you are only allowed to store that sort
of data for a limited amount of time. (Maybe you can only store PII for 24
hours, for instance, and PII is included in your
attributes).
The value for "nifi.bored.yield.duration" can probably be scaled back from "10
millis" to something like "1 millis". This comes
into play when a processor's incoming queue is empty or outgoing queue is full,
for instance. It will wait approximately 10 miliseconds
before checking if the issue has resolved. This results in lower CPU usage, but
it also leads to "artificial latency." For a production
server, "1 millis" is probably just fine.
You may also want to consider changing the values of
"nifi.content.repository.archive.max.retention.period" and
"nifi.content.repository.archive.max.usage.percentage"
When you store the provenance data, it's often helpful to be able to view the
data as it was at that point in the flow. These properties control how
long you keep around this data after you've finished processing it, so that you
can go back and look at it for debugging purposes, etc.
These are probably the most critical things, in terms of performance and
utilization.
Thanks
-Mark
On Feb 13, 2019, at 2:31 PM, Chad Woodhead
<[email protected]<mailto:[email protected]>> wrote:
Mark,
That must be it! I have "nifi.provenance.repository.max.storage.size" = 1 GB.
Just bumped that to 200 GB like you suggested and I can see provenance again.
I've always wondered why my provenance partitions never got very large!
While we're on the subject, are there other settings like this that based on
their default values I could be under-utilizing the resources (storage, mem,
CPU, etc.) I have on my servers dedicated to NiFi?
-Chad
On Wed, Feb 13, 2019 at 1:48 PM Mark Payne
<[email protected]<mailto:[email protected]>> wrote:
Hey Chad,
What do you have for the value of the
"nifi.provenance.repository.max.storage.size" property?
We will often see this if the value is very small (the default is 1 GB, which
is very small) and the volume
of data is reasonably high.
The way that the repo works, it writes to one file for a while, until that file
reaches 100 MB or up to 30 seconds,
by default (configured via "nifi.provenance.repository.rollover.size" and
"nifi.provenance.repository.rollover.time").
At that point, it rolls over to writing to a new file and adds the
now-completed file to a queue. A background thread
is then responsible for compressing that completed file.
What can happen, though, if the max storage space is small is that the data can
actually be aged off from the repository
before that background task attempts to compress it. That can result in either
a FileNotFoundException or an EOFException
when trying to read the TOC file (depending on the timing of when the age-off
happens). It could potentially occur on the
.prov file, in addition to, or instead of the .toc file.
So generally, the solution is to increase the max storage size. It looks like
you have 130 GB on each partition and 2 partitions per
node, so 260 GB total per node that you can use for provenance. So I would set
the max storage size to something like "200 GB".
Since it is a soft limit and it may use more disk space than that temporarily
before shrinking back down, you'll want to give it a little
bit of wiggle room.
Thanks
-Mark
On Feb 13, 2019, at 1:31 PM, Chad Woodhead
<[email protected]<mailto:[email protected]>> wrote:
Hey Joe,
Yes
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
Disk space is fine as well. I have dedicated mounts for provenance (as well all
the repos have their own dedicated mounts):
nifi.provenance.repository.dir.default=/data/disk5/nifi/provenance_repository
nifi.provenance.repository.directory.provenance2=/data/disk6/nifi/provenance_repository
Both of these mounts have plenty of space and are only 1% full and have never
become close to being filled up.
<image.png>
-Chad
On Wed, Feb 13, 2019 at 1:06 PM Joe Witt
<[email protected]<mailto:[email protected]>> wrote:
Chad,
In your conf/nifi.properties please see what the implementation is for your
provenance repository. This specied on
nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository
Is that what you have?
The above error I believe could occur if the location where provenance is being
written runs out of disk space. It is important to ensure that prov is sized
and on a partition alone where this wont happen. This is also true for the
flow file repo. The content repo is more resilient to this by design but still
you want all three repo areas on their own partitions as per best practices.
Thanks
Joe
On Wed, Feb 13, 2019 at 1:03 PM Chad Woodhead
<[email protected]<mailto:[email protected]>> wrote:
I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and I am
seeing the following error in my logs a lot and I can't view any provenance
data in the UI:
2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1]
o.a.n.p.s.EventFileCompressor Failed to read TOC File
/data/disk5/nifi/provenance_repository/toc/158994812.toc
java.io.EOFException: null
at
org.apache.nifi.provenance.toc.StandardTocReader.<init>(StandardTocReader.java:48)
at
org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Any ideas on what could be going on?
-Chad
