I am running this on OpenVMS V8.2-1 on IA64. For a small number of files this works all fine.
I checked the resources part and i have enough disk and ram available. Regards, Rishi Michael McCandless-2 wrote: > > It's very odd that CheckIndex has no trouble opening the segment's > files, yet when you run optimize the OS reports a "file not found" > exception (errno 5). Something odd is happening at the OS/filesystem > level. > > What OS are you running on? > > Can you boil this down to a smallish standalone test that shows the > problem? > > Mike > > On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.ri...@gmail.com> > wrote: >> >> I tried creating the index in different disks but still i see the issue >> :-( >> >> I tried to index documents in other disks also and got the same >> exception. >> >> I also tried >> $ java org.apache.lucene.index.CheckIndex >> /SYS$SYSDEVICE/RISHI/melon_1600/ >> -segment _61 >> NOTE: testing will be more thorough if you run java with >> '-ea:org.apache.lucene...', so assertions are enabled >> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/ >> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene >> 2.4] >> >> Checking only these segments: _61: >> No problems were detected with this index. >> >> Regards, >> Rishi >> >> >> Shai Erera wrote: >>> >>> I noticed the exception is "Caused by: java.io.FileNotFoundException: >>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))" >>> >>> I searched for i/o error (errno:5) and found some information which >>> associates it w/ a more native IO problem, like corrupt file due to >>> system >>> crash etc. >>> >>> Did you experience that? Can you try to create the index somewhere else, >>> or >>> on another drive? >>> >>> Shai >>> >>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal >>> <singhal.ri...@gmail.com>wrote: >>> >>>> >>>> It is a local file system. >>>> >>>> We are using lucene 2.4 and java 1.5 >>>> >>>> Regards, >>>> Rishi >>>> >>>> >>>> Shai Erera wrote: >>>> > >>>> > Is that a local file system, or a network share? >>>> > >>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal >>>> > <singhal.ri...@gmail.com>wrote: >>>> > >>>> >> >>>> >> >>Is there any chance that two writers are open on this directory? >>>> >> No, thats not true. >>>> >> >>>> >> >>something external to Lucene is removing files from the directory. >>>> >> No this also has rare chances as I am the owner of these files and >>>> other >>>> >> then me no one can delete the, :-) >>>> >> >>>> >> Here are all the files in the index directory: >>>> >> segments.gen;1 segments_1.;1 segments_2.;1 _0.fdt;1 >>>> >> _0.fdx;1 _37.cfs;1 _4c.cfs;1 >>>> _61.cfs;1 >>>> >> _9e.cfs;1 _ay.cfs;1 _az.cfs;1 >>>> _b0.cfs;1 >>>> >> _b2.tis;1 _b3.fdt;1 _b4.fdt;1 >>>> >> >>>> >> I see _37 present. >>>> >> >>>> >> Regards, >>>> >> Rishi >>>> >> >>>> >> >>>> >> Michael McCandless-2 wrote: >>>> >> > >>>> >> > Is there any chance that two writers are open on this directory? >>>> Or, >>>> >> > something external to Lucene is removing files from the directory. >>>> >> > >>>> >> > It looks like there were at least two missing files (_37 >>>> >> > >>>> >> > On Thu, Aug 13, 2009 at 5:19 AM, >>>> rishisinghal<singhal.ri...@gmail.com >>>> > >>>> >> > wrote: >>>> >> >> >>>> >> >> Hi, >>>> >> >> >>>> >> >> I am trying to index documents and when all is complete and >>>> optimize >>>> >> is >>>> >> >> called I get >>>> >> >> >>>> >> >> IFD [main]: setInfoStream >>>> >> >> >>>> >> >>>> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@4fced0 >>>> >> >> IW 0 [main]: setInfoStream: >>>> >> >> >>>> >> >>>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600 >>>> >> >> autoCommit=false mergePolicy=org >>>> >> >> .apache.lucene.index.logbytesizemergepol...@92cb49 >>>> >> >> >>>> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@6038e4 >>>> >> >> ramBufferS >>>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1 >>>> >> >> maxFieldLength=10000 >>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0 >>>> >> _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null >>>> >> docStoreOffset=0 >>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD >>>> >> >> ocs=0 numBufDelTerms=0 >>>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 >>>> >> _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 >>>> _9e:c485->_0 >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending] >>>> >> >> IW 0 [main]: CMS: now merge >>>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: CMS: consider merge _61:c245->_0 _9e:c485->_0 >>>> >> _ay:c225->_0 >>>> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >>>> >> >> IW 0 [main]: CMS: launch new thread [Lucene Merge Thread #73] >>>> >> >> IW 0 [main]: CMS: no more merges pending; now return >>>> >> >> IW 0 [Lucene Merge Thread #73]: CMS: merge thread: start >>>> >> >> IW 0 [Lucene Merge Thread #73]: now merge >>>> >> >> merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 >>>> _b0:c5->_0 >>>> >> into >>>> >> >> _b5 [optimize] >>>> >> >> merge=org.apache.lucene.index.mergepolicy$oneme...@3725db >>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0 >>>> >> _ay:c225->_0 >>>> >> >> _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0 >>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize] >>>> >> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge >>>> >> >> Exception in thread "Lucene Merge Thread #73" >>>> >> >> org.apache.lucene.index.MergePolicy$MergeException: >>>> >> >> java.io.FileNotFoundException: /SY >>>> >> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286) >>>> >> >> Caused by: java.io.FileNotFoundException: >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>>> >> >> at java.io.RandomAccessFile.open(Native Method) >>>> >> >> at >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >>>> >> >> at >>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >>>> >> >> at >>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >>>> >> >> at >>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) >>>> >> >> Aug 13, 2009 2:40:56 PM >>>> org.hp.openvms.lucene.index.FileIndexerJSP >>>> >> >> jspMain >>>> >> >> SEVERE: Indexing not completebackground merge hit exception: >>>> >> _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [ >>>> >> >> optimize] >>>> >> >> java.io.IOException: background merge hit exception: _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 >>>> [optimize >>>> >> >> ] >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198) >>>> >> >> at >>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown >>>> >> >> Source) >>>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown >>>> Source) >>>> >> >> Caused by: java.io.FileNotFoundException: >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5)) >>>> >> >> at java.io.RandomAccessFile.open(Native Method) >>>> >> >> at >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321) >>>> >> >> at >>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306) >>>> >> >> at >>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220) >>>> >> >> at >>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260) >>>> >> >> IW 0 [main]: now flush at close >>>> >> >> IW 0 [main]: flush: segment=null docStoreSegment=null >>>> >> docStoreOffset=0 >>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD >>>> >> >> ocs=0 numBufDelTerms=0 >>>> >> >> IW 0 [main]: index before flush _37:c465->_0 _4c:c165->_0 >>>> >> _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: CMS: now merge >>>> >> >> IW 0 [main]: CMS: index: _37:c465->_0 _4c:c165->_0 _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 >>>> >> >> IW 0 [main]: CMS: no more merges pending; now return >>>> >> >> IW 0 [main]: now call final commit() >>>> >> >> IW 0 [main]: startCommit(): start sizeInBytes=0 >>>> >> >> IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0 >>>> _61:c245->_0 >>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11 >>>> >> >> 95 >>>> >> >> IW 0 [main]: now sync _37.cfs >>>> >> >> IW 0 [main]: hit exception while closing >>>> >> >> Exception in thread "main" java.io.FileNotFoundException: >>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5)) >>>> >> >> at java.io.RandomAccessFile.open(Native Method) >>>> >> >> at >>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231) >>>> >> >> at >>>> >> org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456) >>>> >> >> at >>>> >> >> >>>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590) >>>> >> >> at >>>> >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369) >>>> >> >> at >>>> >> >> >>>> >> >>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654) >>>> >> >> at >>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602) >>>> >> >> at >>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578) >>>> >> >> at >>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown >>>> >> >> Source) >>>> >> >> at org.hp.openvms.lucene.index.FileIndexer.main(Unknown >>>> Source) >>>> >> >> >>>> >> >> >>>> >> >> >>>> >> >> I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying >>>> >> >> different >>>> >> >> combinations of these to see if that helps but was unsuccessful. >>>> >> >> >>>> >> >> >>>> >> >> Regards, >>>> >> >> Rishi >>>> >> >> -- >>>> >> >> View this message in context: >>>> >> >> >>>> >> >>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html >>>> >> >> Sent from the Lucene - Java Users mailing list archive at >>>> Nabble.com. >>>> >> >> >>>> >> >> >>>> >> >> >>>> --------------------------------------------------------------------- >>>> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >> >> >>>> >> >> >>>> >> > >>>> >> > >>>> --------------------------------------------------------------------- >>>> >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> >> > For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >> > >>>> >> > >>>> >> > >>>> >> >>>> >> -- >>>> >> View this message in context: >>>> >> >>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html >>>> >> Sent from the Lucene - Java Users mailing list archive at >>>> Nabble.com. >>>> >> >>>> >> >>>> >> >>>> --------------------------------------------------------------------- >>>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >> >>>> >> >>>> > >>>> > >>>> >>>> -- >>>> View this message in context: >>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html >>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>>> >>>> >>> >>> >> >> -- >> View this message in context: >> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > -- View this message in context: http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24958845.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
