Hello Fellas, I am in dire need of your expertise. We have a java process that saves its logs in Lucene indexes. We re-start this process every midnight. While starting, it initializes IndexWriter. We recently had an incident in our production environment when we tried to restart our java process and it failed to start, giving the error:
Unexpected file read error while reading index. > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu"))) > java.lang.RuntimeException: Error attempting to initialize index > /home/maps/appli/expl/lst/sch/Application/prod: > Unexpected file read error while reading index. > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu"))) > at > ... > ... > Caused by: org.apache.lucene.index.CorruptIndexException: Unexpected file > read error while reading index. > > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu"))) > at org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) > ... > ... > Caused by: java.nio.file.NoSuchFileException: > /home/maps/appli/expl/lst/sch/Application/prod/_dko.si at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > ~[?:1.8.0_282] at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > ~[?:1.8.0_282] at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > ~[?:1.8.0_282] at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > ~ > [?:1.8.0_282] at java.nio.channels.FileChannel.open(FileChannel.java:287) > ~[?:1.8.0_282] at > java.nio.channels.FileChannel.open(FileChannel.java:335) ~[?:1.8.0_282] at > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) > ~[lucene-core- > 8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - > 2020-04-08 08:55:42] at > org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) > ~[lucene-core- > 8.5.1.jar:8.5.1 edb9fc409398f2c3446883f9f80595c884d245d0 - ivera - > 2020-04-08 08:55:42] at > org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:9 > ... 4 more We tried to run the CheckIndex command to repair the index, but got the following error while doing so: $ java -cp lucene-core-8.5.1.jar -ea:org.apache.lucene... > org.apache.lucene.index.CheckIndex prod/ -exorcise > > Opening index @ prod/ > > ERROR: could not read any segments file in directory > org.apache.lucene.index.CorruptIndexException: Unexpected file read error > while reading index. > (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/home/maps/appli/expl/lst/sch/Application/prod/segments_2lpu"))) > at > org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:291) > at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:524) > at org.apache.lucene.index.CheckIndex.doCheck(CheckIndex.java:2973) > at org.apache.lucene.index.CheckIndex.doMain(CheckIndex.java:2871) > at org.apache.lucene.index.CheckIndex.main(CheckIndex.java:2797) > Caused by: java.nio.file.NoSuchFileException: > /home/maps/appli/expl/lst/sch/Application/prod/_ogc.si > at > sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) > at > sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) > at > sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177) > at java.nio.channels.FileChannel.open(FileChannel.java:287) > at java.nio.channels.FileChannel.open(FileChannel.java:335) > at > org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:238) > at > org.apache.lucene.store.Directory.openChecksumInput(Directory.java:157) > at > org.apache.lucene.codecs.lucene70.Lucene70SegmentInfoFormat.read(Lucene70SegmentInfoFormat.java:91) > at > org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:353) > at > org.apache.lucene.index.SegmentInfos.readCommit(SegmentInfos.java:289) > ... 4 more For storage of our Lucene indexes, we're using an XFS volume in a SAN (Storage Area Network) (NetApp, full flash, redundant FO connectivity). *OS*: RHEL7 *Lucene version*: 8.5.1 We're not sure how the Lucene indexes got corrupted. Can some expert give us any lead about why this could have happened? In what circumstances can .si files get missing from indexes? We need to know this information so that we can prevent any further occurrence of this issue in our production environment. Thanks so much for your help! Bhavit
