Thanks Mark. I really appreciate the help. I'll take a look at these in my different clusters.
-Chad On Wed, Feb 13, 2019 at 3:09 PM Mark Payne <[email protected]> wrote: > Depending on the size of the nodes, you may want to also increase the > number of indexing threads > ("nifi.provenance.repository.index.threads"). You may want to also > increase the amount of time to store provenance > from 24 hours to something like 36 months... for most people just limiting > based on size is enough. The time is really > only useful if you are worried about it from a compliance standpoint, such > that you are only allowed to store that sort > of data for a limited amount of time. (Maybe you can only store PII for 24 > hours, for instance, and PII is included in your > attributes). > > The value for "nifi.bored.yield.duration" can probably be scaled back from > "10 millis" to something like "1 millis". This comes > into play when a processor's incoming queue is empty or outgoing queue is > full, for instance. It will wait approximately 10 miliseconds > before checking if the issue has resolved. This results in lower CPU > usage, but it also leads to "artificial latency." For a production > server, "1 millis" is probably just fine. > > You may also want to consider changing the values of > "nifi.content.repository.archive.max.retention.period" and > "nifi.content.repository.archive.max.usage.percentage" > When you store the provenance data, it's often helpful to be able to view > the data as it was at that point in the flow. These properties control how > long you keep around this data after you've finished processing it, so > that you can go back and look at it for debugging purposes, etc. > > These are probably the most critical things, in terms of performance and > utilization. > > Thanks > -Mark > > > On Feb 13, 2019, at 2:31 PM, Chad Woodhead <[email protected]> wrote: > > Mark, > > That must be it! I have "nifi.provenance.repository.max.storage.size" = 1 > GB. Just bumped that to 200 GB like you suggested and I can see provenance > again. I've always wondered why my provenance partitions never got very > large! > > While we're on the subject, are there other settings like this that based > on their default values I could be under-utilizing the resources (storage, > mem, CPU, etc.) I have on my servers dedicated to NiFi? > > -Chad > > On Wed, Feb 13, 2019 at 1:48 PM Mark Payne <[email protected]> wrote: > >> Hey Chad, >> >> What do you have for the value of the >> "nifi.provenance.repository.max.storage.size" property? >> We will often see this if the value is very small (the default is 1 GB, >> which is very small) and the volume >> of data is reasonably high. >> >> The way that the repo works, it writes to one file for a while, until >> that file reaches 100 MB or up to 30 seconds, >> by default (configured via "nifi.provenance.repository.rollover.size" and >> "nifi.provenance.repository.rollover.time"). >> At that point, it rolls over to writing to a new file and adds the >> now-completed file to a queue. A background thread >> is then responsible for compressing that completed file. >> >> What can happen, though, if the max storage space is small is that the >> data can actually be aged off from the repository >> before that background task attempts to compress it. That can result in >> either a FileNotFoundException or an EOFException >> when trying to read the TOC file (depending on the timing of when the >> age-off happens). It could potentially occur on the >> .prov file, in addition to, or instead of the .toc file. >> >> So generally, the solution is to increase the max storage size. It looks >> like you have 130 GB on each partition and 2 partitions per >> node, so 260 GB total per node that you can use for provenance. So I >> would set the max storage size to something like "200 GB". >> Since it is a soft limit and it may use more disk space than that >> temporarily before shrinking back down, you'll want to give it a little >> bit of wiggle room. >> >> Thanks >> -Mark >> >> >> On Feb 13, 2019, at 1:31 PM, Chad Woodhead <[email protected]> >> wrote: >> >> Hey Joe, >> >> Yes nifi.provenance.repository.implementation= >> org.apache.nifi.provenance.WriteAheadProvenanceRepository >> >> Disk space is fine as well. I have dedicated mounts for provenance (as >> well all the repos have their own dedicated mounts): >> >> nifi.provenance.repository.dir.default= >> /data/disk5/nifi/provenance_repository >> nifi.provenance.repository.directory.provenance2= >> /data/disk6/nifi/provenance_repository >> >> Both of these mounts have plenty of space and are only 1% full and have >> never become close to being filled up. >> >> <image.png> >> >> -Chad >> >> On Wed, Feb 13, 2019 at 1:06 PM Joe Witt <[email protected]> wrote: >> >>> Chad, >>> >>> In your conf/nifi.properties please see what the implementation is for >>> your provenance repository. This specied on >>> >>> >>> nifi.provenance.repository.implementation=org.apache.nifi.provenance.WriteAheadProvenanceRepository >>> >>> Is that what you have? >>> >>> The above error I believe could occur if the location where provenance >>> is being written runs out of disk space. It is important to ensure that >>> prov is sized and on a partition alone where this wont happen. This is >>> also true for the flow file repo. The content repo is more resilient to >>> this by design but still you want all three repo areas on their own >>> partitions as per best practices. >>> >>> Thanks >>> Joe >>> >>> On Wed, Feb 13, 2019 at 1:03 PM Chad Woodhead <[email protected]> >>> wrote: >>> >>>> I use the org.apache.nifi.provenance.WriteAheadProvenanceRepository and >>>> I am seeing the following error in my logs a lot and I can't view any >>>> provenance data in the UI: >>>> >>>> 2019-02-13 12:57:44,637 ERROR [Compress Provenance Logs-1-thread-1] >>>> o.a.n.p.s.EventFileCompressor Failed to read TOC File >>>> /data/disk5/nifi/provenance_repository/toc/158994812.toc >>>> java.io.EOFException: null >>>> at >>>> org.apache.nifi.provenance.toc.StandardTocReader.<init>(StandardTocReader.java:48) >>>> at >>>> org.apache.nifi.provenance.serialization.EventFileCompressor.run(EventFileCompressor.java:93) >>>> at >>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) >>>> at >>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) >>>> at java.lang.Thread.run(Thread.java:745) >>>> >>>> Any ideas on what could be going on? >>>> >>>> -Chad >>>> >>> >> >
