I've got this one... let me look for that
2015-10-23 09:00:33,625 WARN [Provenance Maintenance Thread-1]
o.a.n.p.PersistentProvenanceRepository
java.io.IOException: No space left on device
at java.io.FileOutputStream.writeBytes(Native Method) ~[na:1.8.0_51]
at java.io.FileOutputStream.write(FileOutputStream.java:326)
~[na:1.8.0_51]
at
org.apache.lucene.store.FSDirectory$FSIndexOutput$1.write(FSDirectory.java:390)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
java.util.zip.CheckedOutputStream.write(CheckedOutputStream.java:73)
~[na:1.8.0_51]
at
java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
~[na:1.8.0_51]
at
org.apache.lucene.store.OutputStreamIndexOutput.writeBytes(OutputStreamIndexOutput.java:51)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.store.DataOutput.writeBytes(DataOutput.java:53)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.BitVector.writeBits(BitVector.java:272)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.BitVector.write(BitVector.java:227)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.codecs.lucene40.Lucene40LiveDocsFormat.writeLiveDocs(Lucene40LiveDocsFormat.java:107)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.ReadersAndUpdates.writeLiveDocs(ReadersAndUpdates.java:326)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:520)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter$ReaderPool.release(IndexWriter.java:505)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.BufferedUpdatesStream.applyDeletesAndUpdates(BufferedUpdatesStream.java:299)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.applyAllDeletesAndUpdates(IndexWriter.java:3312)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:3303)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2989)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:3134)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3101)
~[lucene-core-4.10.4.jar:4.10.4 1662817 - mike - 2015-02-27 16:38:43]
at
org.apache.nifi.provenance.lucene.DeleteIndexAction.execute(DeleteIndexAction.java:66)
~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
at
org.apache.nifi.provenance.PersistentProvenanceRepository.purgeOldEvents(PersistentProvenanceRepository.java:906)
~[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
at
org.apache.nifi.provenance.PersistentProvenanceRepository$2.run(PersistentProvenanceRepository.java:260)
[nifi-persistent-provenance-repository-0.3.0.jar:0.3.0]
On Fri, Oct 23, 2015 at 2:44 PM, Mark Payne <[email protected]> wrote:
> Ryan, Elli,
>
> Do you by chance have any error messages in your logs from the
> FileSystemRepository?
>
> I.e., if you perform:
>
> grep FileSystemRepository logs/*
>
> Do you get anything interesting in there?
>
> Thanks
> -Mark
>
>
> > On Oct 23, 2015, at 2:38 PM, Elli Schwarz
> <[email protected]> wrote:
> >
> > I've been working with Ryan. There appear to be a few issues here:
> >
> > - We upgraded from 0.2.0 to 0.3.0 and it appears that
> content_repository archive is now true by default. In 0.2.0 it was false,
> and the documentation still states it is false by default.
> > - When we ran out of disk space overnight, the problem was solved by
> me simply restarting nifi, and that cleared out the archive by itself.
> >
> > - In order to clear up the archive, I had to set archive to true, and
> set max usage to 1%, and restart nifi. That cleared it up, and then I set
> archive to false and restarted again so we don't run out of space.
> > - Based on the above, it appears that something happened yesterday
> that prevented Nifi from clearing out the archive even though disk usage
> reached 100%. However, restarting nifi apparently enabled it to perform the
> clearing of the archive. So apparently the max usage setting doesn't work
> under some conditions, but we don't know what conditions occurred overnight
> to cause this problem.
> >
> > Thanks!-Elli
> >
> >
> >
> > On Friday, October 23, 2015 2:29 PM, Ryan H <
> [email protected]> wrote:
> >
> >
> >
> > Agree, they concern the archive... although it sounds like there are 2
> > archives?
> >
> > Within the content_repository folder, there are subfolders with the name
> > 'archive' and files inside them.
> >
> > Example:
> > ./nfii/content_repository/837/archive/1445611320767-837
> >
> > Settings:
> > nifi.content.repository.archive.max.retention.period=12 hours
> > nifi.content.repository.archive.max.usage.percentage=50%
> > nifi.content.repository.archive.enabled=true
> >
> > Last night, our server ran out of disk space because the
> content_repository
> > grew too large. Nifi didn't crash, but the log file contained errors
> > saying the disk was full.
> >
> > We're not sure how, but the content_repository did not respect the above
> > settings.
> >
> > We restarted Nifi, and it only then started to remove files, such as:
> > ./nfii/content_repository/837/archive/1445611320767-837
> >
> > We've turned off archiving for now.
> >
> > Ryan
> >
> >
> >
> >
> > On Fri, Oct 23, 2015 at 1:51 PM, Aldrin Piri <[email protected]>
> wrote:
> >
> >> Ryan,
> >>
> >> Those items only concern the archive. Did you have data enqueued in
> >> connections in your flow? If so, these items are not eligible and could
> >> explain why your disk was filled. Otherwise, can you please provide
> some
> >> additional information so we can dig into why this may have arisen.
> >>
> >> Thanks!
> >>
> >> On Fri, Oct 23, 2015 at 10:25 AM, Ryan H <[email protected]>
> >> wrote:
> >>
> >>> I've got the following set:
> >>>
> >>> nifi.content.repository.archive.max.retention.period=12 hours
> >>> nifi.content.repository.archive.max.usage.percentage=50%
> >>> nifi.content.repository.archive.enabled=true
> >>>
> >>> Yet, the content repo filled my disk last night...
> >>>
> >>>
> >>> On Fri, Oct 23, 2015 at 1:16 PM, Aldrin Piri <[email protected]>
> >> wrote:
> >>>
> >>>> Ryan,
> >>>>
> >>>> Those archive folders map to the
> >> nifi.content.repository.archive.enabled
> >>>> property.
> >>>>
> >>>> What this property provides is a retention of files no longer in the
> >>> system
> >>>> for historical context of your flow's processing and the ability for
> >>>> viewing this in conjunction with provenance events as well as allowing
> >>>> replay. The amount of the archive when enabled is bounded by the
> >>>> properties nifi.content.repository.archive.max.retention.period and
> >>>> nifi.content.repository.archive.max.usage.percentage.
> >>>>
> >>>> Additional detail is available in the system properties of our
> >>>> Administration Guide [1]
> >>>>
> >>>> Let us know if you have additional questions.
> >>>>
> >>>> --aldrin
> >>>>
> >>>> [1]
> >>>>
> >>>>
> >>>
> >>
> https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#system_properties
> >>>>
> >>>> On Fri, Oct 23, 2015 at 10:09 AM, Ryan H <[email protected]
> >
> >>>> wrote:
> >>>>
> >>>>> Interesting.. So what would
> >>>>>
> >>>>> ./nfii/content_repository/837/archive/1445611320767-837
> >>>>>
> >>>>> typically be?
> >>>>>
> >>>>> On Fri, Oct 23, 2015 at 12:56 PM, Andrew Grande <
> >>> [email protected]
> >>>>>
> >>>>> wrote:
> >>>>>
> >>>>>> Attachments don't go through, view at imagebin:
> >>>>>> http://ibin.co/2K3SwR0z8yWX
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 10/23/15, 12:52 PM, "Andrew Grande" <[email protected]>
> >>>> wrote:
> >>>>>>
> >>>>>>> Ryan,
> >>>>>>>
> >>>>>>> ./conf/archive is to create a snapshot of your entire flow, not
> >> the
> >>>>>> content repository data. See the attached screenshot (Settings menu
> >>> on
> >>>>> the
> >>>>>> right).
> >>>>>>>
> >>>>>>> Andrew
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 10/23/15, 12:47 PM, "[email protected] on
> >> behalf
> >>>> of
> >>>>>> Ryan H" <[email protected] on behalf of
> >>>>>> [email protected]> wrote:
> >>>>>>>
> >>>>>>>> Hi,
> >>>>>>>> I'm noticing my Content Repo growing large. There's a number
> >> of
> >>>>>> files...
> >>>>>>>>
> >>>>>>>> content_repo/837/archive/144...-837
> >>>>>>>>
> >>>>>>>> Is this new in 3.0? My conf file says any archiving should be
> >>>> going
> >>>>>>>> into ./conf/archive, but i don't see anything in there.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> Ryan
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
> >
> >
>
>
