Hi Fabian, I was wondering of doing the same setup you decribe and found your email. The issue https://bugzilla.redhat.com/show_bug.cgi?id=1390050 was closed as of Oct 2018, have you resolved your issues? What are your experiences with NiFi over GlusterFS? Should I even try or is it worthless. The setup would be very similar NiFi in Docker Swarm with Volumes in GlusterFS.
Thank you. Michal On Wed, Jan 3, 2018 at 4:42 PM Necci, Fabian <[email protected]> wrote: > > Hi all, > > I have one NiFi instance running inside a Docker container and managed by > Kubernetes. > > > > To maintain the application state, the NiFi repositories are under the path > /usr/nifi/data which is a GlusterFS mount. > > > > After sometimes, the lucene index of the provenance repository got corrupted: > > > > 2018-01-03 12:05:07,879 INFO [Provenance Repository Rollover Thread-2] > SimpleIndexManager.java:309 Index Writer for > ./data/provenance_repository/index-1512127508000 has been returned to Index > Manager and is no longer in use. Closing Index Writer > > 2018-01-03 12:05:15,669 INFO [Provenance Repository Rollover Thread-2] > PersistentProvenanceRepository.java:1903 Successfully merged 16 journal files > (5 records) into single Provenance Log File > ./data/provenance_repository/492.prov in 18387 milliseconds > > 2018-01-03 12:05:15,754 INFO [Provenance Repository Rollover Thread-2] > PersistentProvenanceRepository.java:1324 Successfully Rolled over Provenance > Event file containing 29 records. In the past 5 minutes, 33 events have been > written to the Provenance Repository, totaling 178.78 KB > > 2018-01-03 12:05:30,124 INFO [Provenance Maintenance Thread-3] > PersistentProvenanceRepository.java:349 Created new Provenance Event Writers > for events starting with ID 499 > > 2018-01-03 12:05:30,128 INFO [Provenance Maintenance Thread-3] > KyloPersistentProvenanceEventRepository.java:78 Created 16 > KyloRecordWriterDelegate objects starting with: 499 > > 2018-01-03 12:05:33,494 ERROR [Lucene Merge Thread #0] NiFi.java:83 An > Unknown Error Occurred in Thread Thread[Lucene Merge Thread #0,6,main]: > org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: > read past EOF: > MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") > [slice=_ii.fdx] > > 2018-01-03 12:05:33,523 ERROR [Lucene Merge Thread #0] NiFi.java:84 > > org.apache.lucene.index.MergePolicy$MergeException: java.io.EOFException: > read past EOF: > MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") > [slice=_ii.fdx] > > at > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:549) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:522) > > Caused by: java.io.EOFException: read past EOF: > MMapIndexInput(path="/usr/nifi/data/provenance_repository/index-1512127508000/_ii.cfs") > [slice=_ii.fdx] > > at > org.apache.lucene.store.ByteBufferIndexInput.readByte(ByteBufferIndexInput.java:81) > > at > org.apache.lucene.store.BufferedChecksumIndexInput.readByte(BufferedChecksumIndexInput.java:41) > > at org.apache.lucene.store.DataInput.readInt(DataInput.java:98) > > at org.apache.lucene.store.DataInput.readLong(DataInput.java:154) > > at org.apache.lucene.codecs.CodecUtil.checkFooter(CodecUtil.java:209) > > at > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.<init>(CompressingStoredFieldsReader.java:124) > > at > org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113) > > at > org.apache.lucene.index.SegmentCoreReaders.<init>(SegmentCoreReaders.java:133) > > at org.apache.lucene.index.SegmentReader.<init>(SegmentReader.java:108) > > at > org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:145) > > at > org.apache.lucene.index.ReadersAndUpdates.getReaderForMerge(ReadersAndUpdates.java:664) > > at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4152) > > at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3811) > > at > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:409) > > at > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:486) > > > > Has anyboy any idea how could I debug this problem? > > > > I found these issues that could be related: > > https://bugzilla.redhat.com/show_bug.cgi?id=1379568 > > https://bugzilla.redhat.com/show_bug.cgi?id=1390050 > > > > Thanks, > > Fabian
