If it's easily reproducible, can you please provide a test that reproduces it, even if on just your FS? And also, can you turn on infoStream on IndexWriter and post the output?
If CheckIndex succeeds, then I'd compare the output w/ and w/o CheckIndex. Shai On Fri, Aug 14, 2009 at 12:17 AM, rishisinghal <singhal.ri...@gmail.com>wrote: > > I am running this on OpenVMS V8.2-1 on IA64. For a small number of files > this > works all fine. > > I checked the resources part and i have enough disk and ram available. > > Regards, > Rishi > > > Michael McCandless-2 wrote: > > > > It's very odd that CheckIndex has no trouble opening the segment's > > files, yet when you run optimize the OS reports a "file not found" > > exception (errno 5). Something odd is happening at the OS/filesystem > > level. > > > > What OS are you running on? > > > > Can you boil this down to a smallish standalone test that shows the > > problem? > > > > Mike > > > > On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.ri...@gmail.com> > > wrote: > >> > >> I tried creating the index in different disks but still i see the issue > >> :-( > >> > >> I tried to index documents in other disks also and got the same > >> exception. > >> > >> I also tried > >> $ java org.apache.lucene.index.CheckIndex > >> /SYS$SYSDEVICE/RISHI/melon_1600/ > >> -segment _61 > >> NOTE: testing will be more thorough if you run java with > >> '-ea:org.apache.lucene...', so assertions are enabled > >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/ > >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene > >> 2.4] > >> > >> Checking only these segments: _61: > >> No problems were detected with this index. > >> > >> Regards, > >> Rishi > >> > >> > >> Shai Erera wrote: > >>> > >>> I noticed the exception is "Caused by: java.io.FileNotFoundException: > >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))" > >>> > >>> I searched for i/o error (errno:5) and found some information which > >>> associates it w/ a more native IO problem, like corrupt file due to > >>> system > >>> crash etc. > >>> > >>> Did you experience that? Can you try to create the index somewhere > else, > >>> or > >>> on another drive? > >>> > >>> Shai > >>> > >>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal > >>> <singhal.ri...@gmail.com>wrote: > >>> > >>>> > >>>> It is a local file system. > >>>> > >>>> We are using lucene 2.4 and java 1.5 > >>>> > >>>> Regards, > >>>> Rishi > >>>> > >>>> > >>>> Shai Erera wrote: > >>>> > > >>>> > Is that a local file system, or a network share? > >>>> > > >>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal > >>>> > <singhal.ri...@gmail.com>wrote: > >>>> > > >>>> >> > >>>> >> >>Is there any chance that two writers are open on this directory? > >>>> >> No, thats not true. > >>>> >> > >>>> >> >>something external to Lucene is removing files from the > directory. > >>>> >> No this also has rare chances as I am the owner of these files and > >>>> other > >>>> >> then me no one can delete the, :-) > >>>> >> > >>>> >> Here are all the files in the index directory: > >>>> >> segments.gen;1 segments_1.;1 segments_2.;1 > _0.fdt;1 > >>>> >> _0.fdx;1 _37.cfs;1 _4c.cfs;1 > >>>> _61.cfs;1 > >>>> >> _9e.cfs;1 _ay.cfs;1 _az.cfs;1 > >>>> _b0.cfs;1 > >>>> >> _b2.tis;1 _b3.fdt;1 _b4.fdt;1 > >>>> >> > >>>> >> I see _37 present. > >>>> >> > >>>> >> Regards, > >>>> >> Rishi > >>>> >> > >>>> >> > >>>> >> Michael McCandless-2 wrote: > >>>> >> > > >>>> >> > Is there any chance that two writers are open on this directory? > >>>> Or, > >>>> >> > something external to Lucene is removing files from the > directory. > >>>> >> > > >>>> >> > It looks like there were at least two missing files (_37 > >>>> >> > > >>>> >> > On Thu, Aug 13, 2009 at 5:19 AM, > >>>> rishisinghal<singhal.ri...@gmail.com > >>>> > > >>>> >> > wrote: > >>>> >> >> > >>>> >> >> Hi, > >>>> >> >> > >>>> >> >> I am trying to index documents and when all is complete and > >>>> optimize > >>>> >> is > >>>> >> >> called I get > >>>> >> >> > >>>> >> >> IFD [main]: setInfoStream > >>>> >> >> > >>>> >> > >>>> > deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@4fced0 > >>>> >> >> IW 0 [main]: setInfoStream: > >>>> >> >> > >>>> >> > >>>> dir=org.apache.lucene.store.FSDirectory@ > /SYS$SYSDEVICE/RISHI/melon_1600 > >>>> >> >> autoCommit=false mergePolicy=org > >>>> >> >> .apache.lucene.index.logbytesizemergepol...@92cb49 > >>>> >> >> > >>>> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@6038e4 > >>>> >> >> ramBufferS > >>>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 > >>>> >> >> maxFieldLength=10000 > >>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 > >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 > >>>> >> _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null > >>>> >> docStoreOffset=0 > >>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD > >>>> >> >> ocs=0 numBufDelTerms=0 > >>>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 > >>>> >> _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 > >>>> _9e:c485->_0 > >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending] > >>>> >> >> IW 0 [main]: CMS: now merge > >>>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 > _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0 > >>>> >> _ay:c225->_0 > >>>> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize] > >>>> >> >> IW 0 [main]: CMS: launch new thread [Lucene Merge Thread > #73] > >>>> >> >> IW 0 [main]: CMS: no more merges pending; now return > >>>> >> >> IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start > >>>> >> >> IW 0 [Lucene Merge Thread #73]: now merge > >>>> >> >> merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 > >>>> _b0:c5->_0 > >>>> >> into > >>>> >> >> _b5 [optimize] > >>>> >> >> merge=org.apache.lucene.index.mergepolicy$oneme...@3725db > >>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 > >>>> >> _ay:c225->_0 > >>>> >> >> _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 > _9e:c485->_0 > >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize] > >>>> >> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge > >>>> >> >> Exception in thread "Lucene Merge Thread #73" > >>>> >> >> org.apache.lucene.index.MergePolicy$MergeException: > >>>> >> >> java.io.FileNotFoundException: /SY > >>>> >> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286) > >>>> >> >> Caused by: java.io.FileNotFoundException: > >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) > >>>> >> >> at java.io.RandomAccessFile.open(Native Method) > >>>> >> >> at > >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) > >>>> >> >> at > >>>> >> >> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) > >>>> >> >> at > >>>> >> >> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) > >>>> >> >> at > >>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) > >>>> >> >> Aug 13, 2009 2:40:56 PM > >>>> org.hp.openvms.lucene.index.FileIndexerJSP > >>>> >> >> jspMain > >>>> >> >> SEVERE: Indexing not completebackground merge hit exception: > >>>> >> _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [ > >>>> >> >> optimize] > >>>> >> >> java.io.IOException: background merge hit exception: > _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 > >>>> [optimize > >>>> >> >> ] > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198) > >>>> >> >> at > >>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown > >>>> >> >> Source) > >>>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown > >>>> Source) > >>>> >> >> Caused by: java.io.FileNotFoundException: > >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) > >>>> >> >> at java.io.RandomAccessFile.open(Native Method) > >>>> >> >> at > >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) > >>>> >> >> at > >>>> >> >> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) > >>>> >> >> at > >>>> >> >> > org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) > >>>> >> >> at > >>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) > >>>> >> >> IW 0 [main]: now flush at close > >>>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null > >>>> >> docStoreOffset=0 > >>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD > >>>> >> >> ocs=0 numBufDelTerms=0 > >>>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 > >>>> >> _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: CMS: now merge > >>>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 > _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 > >>>> >> >> IW 0 [main]: CMS: no more merges pending; now return > >>>> >> >> IW 0 [main]: now call final commit() > >>>> >> >> IW 0 [main]: startCommit(): start sizeInBytes=0 > >>>> >> >> IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 > >>>> _61:c245->_0 > >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11 > >>>> >> >> 95 > >>>> >> >> IW 0 [main]: now sync _37.cfs > >>>> >> >> IW 0 [main]: hit exception while closing > >>>> >> >> Exception in thread "main" java.io.FileNotFoundException: > >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5)) > >>>> >> >> at java.io.RandomAccessFile.open(Native Method) > >>>> >> >> at > >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) > >>>> >> >> at > >>>> >> org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456) > >>>> >> >> at > >>>> >> >> > >>>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590) > >>>> >> >> at > >>>> >> >> > org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369) > >>>> >> >> at > >>>> >> >> > >>>> >> > >>>> > org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654) > >>>> >> >> at > >>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602) > >>>> >> >> at > >>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578) > >>>> >> >> at > >>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown > >>>> >> >> Source) > >>>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown > >>>> Source) > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> >> >> I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying > >>>> >> >> different > >>>> >> >> combinations of these to see if that helps but was unsuccessful. > >>>> >> >> > >>>> >> >> > >>>> >> >> Regards, > >>>> >> >> Rishi > >>>> >> >> -- > >>>> >> >> View this message in context: > >>>> >> >> > >>>> >> > >>>> > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html > >>>> >> >> Sent from the Lucene - Java Users mailing list archive at > >>>> Nabble.com. > >>>> >> >> > >>>> >> >> > >>>> >> >> > >>>> --------------------------------------------------------------------- > >>>> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> >> >> For additional commands, e-mail: > java-user-h...@lucene.apache.org > >>>> >> >> > >>>> >> >> > >>>> >> > > >>>> >> > > >>>> --------------------------------------------------------------------- > >>>> >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> >> > For additional commands, e-mail: > java-user-h...@lucene.apache.org > >>>> >> > > >>>> >> > > >>>> >> > > >>>> >> > >>>> >> -- > >>>> >> View this message in context: > >>>> >> > >>>> > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html > >>>> >> Sent from the Lucene - Java Users mailing list archive at > >>>> Nabble.com. > >>>> >> > >>>> >> > >>>> >> > >>>> --------------------------------------------------------------------- > >>>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>> >> > >>>> >> > >>>> > > >>>> > > >>>> > >>>> -- > >>>> View this message in context: > >>>> > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html > >>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. > >>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org > >>>> > >>>> > >>> > >>> > >> > >> -- > >> View this message in context: > >> > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html > >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. > >> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > >> For additional commands, e-mail: java-user-h...@lucene.apache.org > >> > >> > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > > > > > > -- > View this message in context: > http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
