Here's our provenance settings: # Persistent Provenance Repository Properties nifi.provenance.repository.directory.default=./provenance_repository nifi.provenance.repository.max.storage.time=24 hours nifi.provenance.repository.max.storage.size=1 GB nifi.provenance.repository.rollover.time=30 secs nifi.provenance.repository.rollover.size=100 MB nifi.provenance.repository.query.threads=2 nifi.provenance.repository.index.threads=1 nifi.provenance.repository.compress.on.rollover=true nifi.provenance.repository.always.sync=false nifi.provenance.repository.journal.count=16 # Comma-separated list of fields. Fields that are not indexed will not be searchable. Valid fields are: # EventType, FlowFileUUID, Filename, TransitURI, ProcessorID, AlternateIdentifierURI, ContentType, Relationship, Details nifi.provenance.repository.indexed.fields=EventType, FlowFileUUID, Filename, ProcessorID, Relationship # FlowFile Attributes that should be indexed and made searchable nifi.provenance.repository.indexed.attributes= # Large values for the shard size will result in more Java heap usage when searching the Provenance Repository # but should provide better performance nifi.provenance.repository.index.shard.size=500 MB # Indicates the maximum length that a FlowFile attribute can be when retrieving a Provenance Event from # the repository. If the length of any attribute exceeds this value, it will be truncated when the event is retrieved. nifi.provenance.repository.max.attribute.length=65536
# Volatile Provenance Respository Properties nifi.provenance.repository.buffer.size=100000 On Fri, Oct 23, 2015 at 2:57 PM, Elli Schwarz < [email protected]> wrote: > We had max storage size of 1GB, but that's for provenance repo and our > problem was with content_repo. Our disk was 60GB, all on one partition, and > 55GB were taken up by content_repo. Now, it only contains 233MB. > > > > On Friday, October 23, 2015 2:50 PM, Mark Payne <[email protected]> > wrote: > > > > OK, so this is interesting. Do you have your content repository and > provenance repository > both pointing to the same partition? What do you have the > "nifi.provenance.repository.max.storage.size" > property set to? How large is the actual disk? > > Thanks > -Mark > > > > On Oct 23, 2015, at 2:45 PM, Ryan H <[email protected]> wrote: > > > > I've got this one... let me look for that > > > > 2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1] > > o.a.n.p.PersistentProvenanceRepository > > java.io.IOException: No space left on device > > at java.io.FileOutputStream.writeBytes(Native Method) > ~[na:1.8.0_51] > > at java.io.FileOutputStream.write(FileOutputStream.java:326) > > ~[na:1.8.0_51] > > at > > > org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73) > > ~[na:1.8.0_51] > > at > > java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) > > ~[na:1.8.0_51] > > at > > > org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101) > > ~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43] > > at > > > org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66) > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > at > > > org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906) > > ~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > at > > > org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260) > > [nifi-persistent-provenance-repository-0.3.0.jar:0.3.0] > > > > On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <[email protected]> > wrote: > > > >> Ryan, Elli, > >> > >> Do you by chance have any error messages in your logs from the > >> FileSystemRepository? > >> > >> I.e., if you perform: > >> > >> grep FileSystemRepository logs/* > >> > >> Do you get anything interesting in there? > >> > >> Thanks > >> -Mark > >> > >> > >>> On Oct 23, 2015, at 2:38 PM, Elli Schwarz > >> <[email protected]> wrote: > >>> > >>> I've been working with Ryan. There appear to be a few issues here: > >>> > >>> - We upgraded from 0.2.0 to 0.3.0 and it appears that > >> content_repository archive is now true by default. In 0.2.0 it was > false, > >> and the documentation still states it is false by default. > >>> - When we ran out of disk space overnight, the problem was solved by > >> me simply restarting nifi, and that cleared out the archive by itself. > >>> > >>> - In order to clear up the archive, I had to set archive to true, and > >> set max usage to 1%, and restart nifi. That cleared it up, and then I > set > >> archive to false and restarted again so we don't run out of space. > >>> - Based on the above, it appears that something happened yesterday > >> that prevented Nifi from clearing out the archive even though disk usage > >> reached 100%. However, restarting nifi apparently enabled it to perform > the > >> clearing of the archive. So apparently the max usage setting doesn't > work > >> under some conditions, but we don't know what conditions occurred > overnight > >> to cause this problem. > >>> > >>> Thanks!-Elli > >>> > >>> > >>> > >>> On Friday, October 23, 2015 2:29 PM, Ryan H < > >> [email protected]> wrote: > >>> > >>> > >>> > >>> Agree, they concern the archive... although it sounds like there are 2 > >>> archives? > >>> > >>> Within the content_repository folder, there are subfolders with the > name > >>> 'archive' and files inside them. > >>> > >>> Example: > >>> ./nfii/content_repository/837/archive/1445611320767-837 > >>> > >>> Settings: > >>> nifi.content.repository.archive.max.retention.period=12 hours > >>> nifi.content.repository.archive.max.usage.percentage=50% > >>> nifi.content.repository.archive.enabled=true > >>> > >>> Last night, our server ran out of disk space because the > >> content_repository > >>> grew too large. Nifi didn't crash, but the log file contained errors > >>> saying the disk was full. > >>> > >>> We're not sure how, but the content_repository did not respect the > above > >>> settings. > >>> > >>> We restarted Nifi, and it only then started to remove files, such as: > >>> ./nfii/content_repository/837/archive/1445611320767-837 > >>> > >>> We've turned off archiving for now. > >>> > >>> Ryan > >>> > >>> > >>> > >>> > >>> On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <[email protected]> > >> wrote: > >>> > >>>> Ryan, > >>>> > >>>> Those items only concern the archive. Did you have data enqueued in > >>>> connections in your flow? If so, these items are not eligible and > could > >>>> explain why your disk was filled. Otherwise, can you please provide > >> some > >>>> additional information so we can dig into why this may have arisen. > >>>> > >>>> Thanks! > >>>> > >>>> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <[email protected] > > > >>>> wrote: > >>>> > >>>>> I've got the following set: > >>>>> > >>>>> nifi.content.repository.archive.max.retention.period=12 hours > >>>>> nifi.content.repository.archive.max.usage.percentage=50% > >>>>> nifi.content.repository.archive.enabled=true > >>>>> > >>>>> Yet, the content repo filled my disk last night... > >>>>> > >>>>> > >>>>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Ryan, > >>>>>> > >>>>>> Those archive folders map to the > >>>> nifi.content.repository.archive.enabled > >>>>>> property. > >>>>>> > >>>>>> What this property provides is a retention of files no longer in the > >>>>> system > >>>>>> for historical context of your flow's processing and the ability for > >>>>>> viewing this in conjunction with provenance events as well as > allowing > >>>>>> replay. The amount of the archive when enabled is bounded by the > >>>>>> properties nifi.content.repository.archive.max.retention.period and > >>>>>> nifi.content.repository.archive.max.usage.percentage. > >>>>>> > >>>>>> Additional detail is available in the system properties of our > >>>>>> Administration Guide [1] > >>>>>> > >>>>>> Let us know if you have additional questions. > >>>>>> > >>>>>> --aldrin > >>>>>> > >>>>>> [1] > >>>>>> > >>>>>> > >>>>> > >>>> > >> > https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties > >>>>>> > >>>>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H < > [email protected] > >>> > >>>>>> wrote: > >>>>>> > >>>>>>> Interesting.. So what would > >>>>>>> > >>>>>>> ./nfii/content_repository/837/archive/1445611320767-837 > >>>>>>> > >>>>>>> typically be? > >>>>>>> > >>>>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande < > >>>>> [email protected] > >>>>>>> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Attachments don't go through, view at imagebin: > >>>>>>>> http://ibin.co/2K3SwR0z8yWX > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <[email protected]> > >>>>>> wrote: > >>>>>>>> > >>>>>>>>> Ryan, > >>>>>>>>> > >>>>>>>>> ./conf/archive is to create a snapshot of your entire flow, not > >>>> the > >>>>>>>> content repository data. See the attached screenshot (Settings > menu > >>>>> on > >>>>>>> the > >>>>>>>> right). > >>>>>>>>> > >>>>>>>>> Andrew > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On 10/23/15, 12:47 PM, "[email protected] on > >>>> behalf > >>>>>> of > >>>>>>>> Ryan H" <[email protected] on behalf of > >>>>>>>> [email protected]> wrote: > >>>>>>>>> > >>>>>>>>>> Hi, > >>>>>>>>>> I'm noticing my Content Repo growing large. There's a number > >>>> of > >>>>>>>> files... > >>>>>>>>>> > >>>>>>>>>> content_repo/837/archive/144...-837 > >>>>>>>>>> > >>>>>>>>>> Is this new in 3.0? My conf file says any archiving should be > >>>>>> going > >>>>>>>>>> into ./conf/archive, but i don't see anything in there. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> Ryan > >>>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>> > >>> > >>> > >>> > >> > >> > > > > >
