Mark, That must be it! I have "nifi.provenance.repository.max.storage.size" = 1 GB. Just bumped that to 200 GB like you suggested and I can see provenance again. I've always wondered why my provenance partitions never got very large!
While we're on the subject, are there other settings like this that based on their default values I could be under-utilizing the resources (storage, mem, CPU, etc.) I have on my servers dedicated to NiFi? -Chad On Wed, Feb 13, 2019 at 1:48 PM Mark Payne <[email protected]> wrote: > Hey Chad, > > What do you have for the value of the > "nifi.provenance.repository.max.storage.size" property? > We will often see this if the value is very small (the default is 1 GB, > which is very small) and the volume > of data is reasonably high. > > The way that the repo works, it writes to one file for a while, until that > file reaches 100 MB or up to 30 seconds, > by default (configured via "nifi.provenance.repository.rollover.size" and > "nifi.provenance.repository.rollover.time"). > At that point, it rolls over to writing to a new file and adds the > now-completed file to a queue. A background thread > is then responsible for compressing that completed file. > > What can happen, though, if the max storage space is small is that the > data can actually be aged off from the repository > before that background task attempts to compress it. That can result in > either a FileNotFoundException or an EOFException > when trying to read the TOC file (depending on the timing of when the > age-off happens). It could potentially occur on the > .prov file, in addition to, or instead of the .toc file. > > So generally, the solution is to increase the max storage size. It looks > like you have 130 GB on each partition and 2 partitions per > node, so 260 GB total per node that you can use for provenance. So I would > set the max storage size to something like "200 GB". > Since it is a soft limit and it may use more disk space than that > temporarily before shrinking back down, you'll want to give it a little > bit of wiggle room. > > Thanks > -Mark > > > On Feb 13, 2019, at 1:31 PM, Chad Woodhead <[email protected]> wrote: > > Hey Joe, > > Yes nifi.provenance.repository.implementation= > org.apache.nifi.provenance.WriteAheadProvenanceRepository > > Disk space is fine as well. I have dedicated mounts for provenance (as > well all the repos have their own dedicated mounts): > > nifi.provenance.repository.dir.default= > /data/disk5/nifi/provenance_repository > nifi.provenance.repository.directory.provenance2= > /data/disk6/nifi/provenance_repository > > Both of these mounts have plenty of space and are only 1% full and have > never become close to being filled up. > > <image.png> > > -Chad > > On Wed, Feb 13, 2019 at 1:06 PM Joe Witt <[email protected]> wrote: > >> Chad, >> >> In your conf/nifi.properties please see what the implementation is for >> your provenance repository. This specied on >> >> >> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository >> >> Is that what you have? >> >> The above error I believe could occur if the location where provenance is >> being written runs out of disk space. It is important to ensure that prov >> is sized and on a partition alone where this wont happen. This is also >> true for the flow file repo. The content repo is more resilient to this by >> design but still you want all three repo areas on their own partitions as >> per best practices. >> >> Thanks >> Joe >> >> On Wed, Feb 13, 2019 at 1:03 PM Chad Woodhead <[email protected]> >> wrote: >> >>> I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and >>> I am seeing the following error in my logs a lot and I can't view any >>> provenance data in the UI: >>> >>> 2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1] >>> o.a.n.p.s.EventFileCompressor Failed to read TOC File >>> /data/disk5/nifi/provenance_repository/toc/158994812.toc >>> java.io.EOFException: null >>> at >>> org.apache.nifi.provenance.toc.StandardTocReader.<init>(StandardTocReader.java:48) >>> at >>> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>> at java.lang.Thread.run(Thread.java:745) >>> >>> Any ideas on what could be going on? >>> >>> -Chad >>> >> >
