It's very odd that CheckIndex has no trouble opening the segment's files, yet when you run optimize the OS reports a "file not found" exception (errno 5). Something odd is happening at the OS/filesystem level.
What OS are you running on? Can you boil this down to a smallish standalone test that shows the problem? Mike On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.ri...@gmail.com> wrote: > > I tried creating the index in different disks but still i see the issue :-( > > I tried to index documents in other disks also and got the same exception. > > I also tried > $ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/ > -segment _61 > NOTE: testing will be more thorough if you run java with > '-ea:org.apache.lucene...', so assertions are enabled > Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/ > Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4] > > Checking only these segments: _61: > No problems were detected with this index. > > Regards, > Rishi > > > Shai Erera wrote: >> >> I noticed the exception is "Caused by: java.io.FileNotFoundException: >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))" >> >> I searched for i/o error (errno:5) and found some information which >> associates it w/ a more native IO problem, like corrupt file due to system >> crash etc. >> >> Did you experience that? Can you try to create the index somewhere else, >> or >> on another drive? >> >> Shai >> >> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal >> <singhal.ri...@gmail.com>wrote: >> >>> >>> It is a local file system. >>> >>> We are using lucene 2.4 and java 1.5 >>> >>> Regards, >>> Rishi >>> >>> >>> Shai Erera wrote: >>> > >>> > Is that a local file system, or a network share? >>> > >>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal >>> > <singhal.ri...@gmail.com>wrote: >>> > >>> >> >>> >> >>Is there any chance that two writers are open on this directory? >>> >> No, thats not true. >>> >> >>> >> >>something external to Lucene is removing files from the directory. >>> >> No this also has rare chances as I am the owner of these files and >>> other >>> >> then me no one can delete the, :-) >>> >> >>> >> Here are all the files in the index directory: >>> >> segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1 >>> >> _0.fdx;1 _37.cfs;1 _4c.cfs;1 _61.cfs;1 >>> >> _9e.cfs;1 _ay.cfs;1 _az.cfs;1 _b0.cfs;1 >>> >> _b2.tis;1 _b3.fdt;1 _b4.fdt;1 >>> >> >>> >> I see _37 present. >>> >> >>> >> Regards, >>> >> Rishi >>> >> >>> >> >>> >> Michael McCandless-2 wrote: >>> >> > >>> >> > Is there any chance that two writers are open on this directory? >>> Or, >>> >> > something external to Lucene is removing files from the directory. >>> >> > >>> >> > It looks like there were at least two missing files (_37 >>> >> > >>> >> > On Thu, Aug 13, 2009 at 5:19 AM, >>> rishisinghal<singhal.ri...@gmail.com >>> > >>> >> > wrote: >>> >> >> >>> >> >> Hi, >>> >> >> >>> >> >> I am trying to index documents and when all is complete and >>> optimize >>> >> is >>> >> >> called I get >>> >> >> >>> >> >> IFD [main]: setInfoStream >>> >> >> >>> >> >>> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@4fced0 >>> >> >> IW 0 [main]: setInfoStream: >>> >> >> >>> >> >>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600 >>> >> >> autoCommit=false mergePolicy=org >>> >> >> .apache.lucene.index.logbytesizemergepol...@92cb49 >>> >> >> >>> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@6038e4 >>> >> >> ramBufferS >>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 >>> >> >> maxFieldLength=10000 >>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 >>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 >>> >> _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD >>> >> >> ocs=0 numBufDelTerms=0 >>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 >>> >> _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0 >>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending] >>> >> >> IW 0 [main]: CMS: now merge >>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0 >>> >> _ay:c225->_0 >>> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >>> >> >> IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73] >>> >> >> IW 0 [main]: CMS: no more merges pending; now return >>> >> >> IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start >>> >> >> IW 0 [Lucene Merge Thread #73]: now merge >>> >> >> merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> into >>> >> >> _b5 [optimize] >>> >> >> merge=org.apache.lucene.index.mergepolicy$oneme...@3725db >>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 >>> >> _ay:c225->_0 >>> >> >> _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0 >>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >>> >> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge >>> >> >> Exception in thread "Lucene Merge Thread #73" >>> >> >> org.apache.lucene.index.MergePolicy$MergeException: >>> >> >> java.io.FileNotFoundException: /SY >>> >> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286) >>> >> >> Caused by: java.io.FileNotFoundException: >>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>> >> >> at java.io.RandomAccessFile.open(Native Method) >>> >> >> at >>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) >>> >> >> at >>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >>> >> >> at >>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >>> >> >> at >>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >>> >> >> at >>> >> >> >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) >>> >> >> Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP >>> >> >> jspMain >>> >> >> SEVERE: Indexing not completebackground merge hit exception: >>> >> _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [ >>> >> >> optimize] >>> >> >> java.io.IOException: background merge hit exception: _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize >>> >> >> ] >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198) >>> >> >> at >>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown >>> >> >> Source) >>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown >>> Source) >>> >> >> Caused by: java.io.FileNotFoundException: >>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>> >> >> at java.io.RandomAccessFile.open(Native Method) >>> >> >> at >>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) >>> >> >> at >>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >>> >> >> at >>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >>> >> >> at >>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >>> >> >> at >>> >> >> >>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) >>> >> >> IW 0 [main]: now flush at close >>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null >>> >> docStoreOffset=0 >>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD >>> >> >> ocs=0 numBufDelTerms=0 >>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 >>> >> _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: CMS: now merge >>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>> >> >> IW 0 [main]: CMS: no more merges pending; now return >>> >> >> IW 0 [main]: now call final commit() >>> >> >> IW 0 [main]: startCommit(): start sizeInBytes=0 >>> >> >> IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 >>> _61:c245->_0 >>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11 >>> >> >> 95 >>> >> >> IW 0 [main]: now sync _37.cfs >>> >> >> IW 0 [main]: hit exception while closing >>> >> >> Exception in thread "main" java.io.FileNotFoundException: >>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5)) >>> >> >> at java.io.RandomAccessFile.open(Native Method) >>> >> >> at >>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>> >> >> at >>> >> org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456) >>> >> >> at >>> >> >> >>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369) >>> >> >> at >>> >> >> >>> >> >>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602) >>> >> >> at >>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578) >>> >> >> at >>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown >>> >> >> Source) >>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown >>> Source) >>> >> >> >>> >> >> >>> >> >> >>> >> >> I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying >>> >> >> different >>> >> >> combinations of these to see if that helps but was unsuccessful. >>> >> >> >>> >> >> >>> >> >> Regards, >>> >> >> Rishi >>> >> >> -- >>> >> >> View this message in context: >>> >> >> >>> >> >>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html >>> >> >> Sent from the Lucene - Java Users mailing list archive at >>> Nabble.com. >>> >> >> >>> >> >> >>> >> >> >>> --------------------------------------------------------------------- >>> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >> >>> >> >> >>> >> > >>> >> > >>> --------------------------------------------------------------------- >>> >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> > For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> > >>> >> > >>> >> > >>> >> >>> >> -- >>> >> View this message in context: >>> >> >>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html >>> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>> >> >>> >> >>> >> --------------------------------------------------------------------- >>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >> >>> >> >>> > >>> > >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html >>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >> >> > > -- > View this message in context: > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org