Hey Chad, What do you have for the value of the "nifi.provenance.repository.max.storage.size" property? We will often see this if the value is very small (the default is 1 GB, which is very small) and the volume of data is reasonably high.
The way that the repo works, it writes to one file for a while, until that file reaches 100 MB or up to 30 seconds, by default (configured via "nifi.provenance.repository.rollover.size" and "nifi.provenance.repository.rollover.time"). At that point, it rolls over to writing to a new file and adds the now-completed file to a queue. A background thread is then responsible for compressing that completed file. What can happen, though, if the max storage space is small is that the data can actually be aged off from the repository before that background task attempts to compress it. That can result in either a FileNotFoundException or an EOFException when trying to read the TOC file (depending on the timing of when the age-off happens). It could potentially occur on the .prov file, in addition to, or instead of the .toc file. So generally, the solution is to increase the max storage size. It looks like you have 130 GB on each partition and 2 partitions per node, so 260 GB total per node that you can use for provenance. So I would set the max storage size to something like "200 GB". Since it is a soft limit and it may use more disk space than that temporarily before shrinking back down, you'll want to give it a little bit of wiggle room. Thanks -Mark On Feb 13, 2019, at 1:31 PM, Chad Woodhead <[email protected]<mailto:[email protected]>> wrote: Hey Joe, Yes nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository Disk space is fine as well. I have dedicated mounts for provenance (as well all the repos have their own dedicated mounts): nifi.provenance.repository.dir.default=/data/disk5/nifi/provenance_repository nifi.provenance.repository.directory.provenance2=/data/disk6/nifi/provenance_repository Both of these mounts have plenty of space and are only 1% full and have never become close to being filled up. <image.png> -Chad On Wed, Feb 13, 2019 at 1:06 PM Joe Witt <[email protected]<mailto:[email protected]>> wrote: Chad, In your conf/nifi.properties please see what the implementation is for your provenance repository. This specied on nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository Is that what you have? The above error I believe could occur if the location where provenance is being written runs out of disk space. It is important to ensure that prov is sized and on a partition alone where this wont happen. This is also true for the flow file repo. The content repo is more resilient to this by design but still you want all three repo areas on their own partitions as per best practices. Thanks Joe On Wed, Feb 13, 2019 at 1:03 PM Chad Woodhead <[email protected]<mailto:[email protected]>> wrote: I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and I am seeing the following error in my logs a lot and I can't view any provenance data in the UI: 2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1] o.a.n.p.s.EventFileCompressor Failed to read TOC File /data/disk5/nifi/provenance_repository/toc/158994812.toc java.io.EOFException: null at org.apache.nifi.provenance.toc.StandardTocReader.<init>(StandardTocReader.java:48) at org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Any ideas on what could be going on? -Chad
