It's very odd that CheckIndex has no trouble opening the segment's
files, yet when you run optimize the OS reports a "file not found"
exception (errno 5).  Something odd is happening at the OS/filesystem
level.

What OS are you running on?

Can you boil this down to a smallish standalone test that shows the problem?

Mike

On Thu, Aug 13, 2009 at 8:21 AM, rishisinghal<singhal.ri...@gmail.com> wrote:
>
> I tried creating the index in different disks but still i see the issue :-(
>
> I tried to index documents in other disks also and got the same exception.
>
> I also tried
> $ java org.apache.lucene.index.CheckIndex /SYS$SYSDEVICE/RISHI/melon_1600/
> -segment _61
> NOTE: testing will be more thorough if you run java with
> '-ea:org.apache.lucene...', so assertions are enabled
> Opening index @ /SYS$SYSDEVICE/RISHI/melon_1600/
> Segments file=segments_1 numSegments=0 version=FORMAT_HAS_PROX [Lucene 2.4]
>
> Checking only these segments: _61:
> No problems were detected with this index.
>
> Regards,
> Rishi
>
>
> Shai Erera wrote:
>>
>> I noticed the exception is "Caused by: java.io.FileNotFoundException:
>> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))"
>>
>> I searched for i/o error (errno:5) and found some information which
>> associates it w/ a more native IO problem, like corrupt file due to system
>> crash etc.
>>
>> Did you experience that? Can you try to create the index somewhere else,
>> or
>> on another drive?
>>
>> Shai
>>
>> On Thu, Aug 13, 2009 at 3:00 PM, rishisinghal
>> <singhal.ri...@gmail.com>wrote:
>>
>>>
>>> It is a local file system.
>>>
>>> We are using lucene 2.4 and java 1.5
>>>
>>> Regards,
>>> Rishi
>>>
>>>
>>> Shai Erera wrote:
>>> >
>>> > Is that a local file system, or a network share?
>>> >
>>> > On Thu, Aug 13, 2009 at 1:07 PM, rishisinghal
>>> > <singhal.ri...@gmail.com>wrote:
>>> >
>>> >>
>>> >> >>Is there any chance that two writers are open on this directory?
>>> >> No, thats not true.
>>> >>
>>> >> >>something external to Lucene is removing files from the directory.
>>> >> No this also has rare chances as I am the owner of these files and
>>> other
>>> >> then me no one can delete the, :-)
>>> >>
>>> >> Here are all the files in the index directory:
>>> >> segments.gen;1      segments_1.;1       segments_2.;1       _0.fdt;1
>>> >> _0.fdx;1            _37.cfs;1           _4c.cfs;1           _61.cfs;1
>>> >> _9e.cfs;1           _ay.cfs;1           _az.cfs;1           _b0.cfs;1
>>> >> _b2.tis;1           _b3.fdt;1           _b4.fdt;1
>>> >>
>>> >> I see _37 present.
>>> >>
>>> >> Regards,
>>> >> Rishi
>>> >>
>>> >>
>>> >> Michael McCandless-2 wrote:
>>> >> >
>>> >> > Is there any chance that two writers are open on this directory?
>>> Or,
>>> >> > something external to Lucene is removing files from the directory.
>>> >> >
>>> >> > It looks like there were at least two missing files (_37
>>> >> >
>>> >> > On Thu, Aug 13, 2009 at 5:19 AM,
>>> rishisinghal<singhal.ri...@gmail.com
>>> >
>>> >> > wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> I am trying to index documents and when all is complete and
>>> optimize
>>> >> is
>>> >> >> called I get
>>> >> >>
>>> >> >> IFD [main]: setInfoStream
>>> >> >>
>>> >>
>>> deletionpolicy=org.apache.lucene.index.keeponlylastcommitdeletionpol...@4fced0
>>> >> >> IW 0 [main]: setInfoStream:
>>> >> >>
>>> >>
>>> dir=org.apache.lucene.store.FSDirectory@/SYS$SYSDEVICE/RISHI/melon_1600
>>> >> >> autoCommit=false mergePolicy=org
>>> >> >> .apache.lucene.index.logbytesizemergepol...@92cb49
>>> >> >>
>>> mergescheduler=org.apache.lucene.index.concurrentmergeschedu...@6038e4
>>> >> >> ramBufferS
>>> >> >> izeMB=16.0 maxBufferedDocs=5 maxBuffereDeleteTerms=-1
>>> >> >> maxFieldLength=10000
>>> >> >> index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>>> >> >>  _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: optimize: index now _37:c465->_0 _4c:c165->_0
>>> >> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>>> >> >> ocs=0 numBufDelTerms=0
>>> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>>> >> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: add merge to pendingMerges: _61:c245->_0 _9e:c485->_0
>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 [optimize] [total 1 pending]
>>> >> >> IW 0 [main]: CMS: now merge
>>> >> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: CMS:   consider merge _61:c245->_0 _9e:c485->_0
>>> >> _ay:c225->_0
>>> >> >> _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>>> >> >> IW 0 [main]: CMS:     launch new thread [Lucene Merge Thread #73]
>>> >> >> IW 0 [main]: CMS:   no more merges pending; now return
>>> >> >> IW 0 [Lucene Merge Thread #73]: CMS:   merge thread: start
>>> >> >> IW 0 [Lucene Merge Thread #73]: now merge
>>> >> >>  merge=_61:c245->_0 _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> into
>>> >> >> _b5 [optimize]
>>> >> >>  merge=org.apache.lucene.index.mergepolicy$oneme...@3725db
>>> >> >>  index=_37:c465->_0 _4c:c165->_0 _61:c245->_0 _9e:c485->_0
>>> >> _ay:c225->_0
>>> >> >> _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [Lucene Merge Thread #73]: merging _61:c245->_0 _9e:c485->_0
>>> >> >> _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize]
>>> >> >> IW 0 [Lucene Merge Thread #73]: hit exception during merge
>>> >> >> Exception in thread "Lucene Merge Thread #73"
>>> >> >> org.apache.lucene.index.MergePolicy$MergeException:
>>> >> >> java.io.FileNotFoundException: /SY
>>> >> >> S$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:309)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:286)
>>> >> >> Caused by: java.io.FileNotFoundException:
>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>>> >> >>        at
>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
>>> >> >>        at
>>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
>>> >> >>        at
>>> >> >>
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
>>> >> >> Aug 13, 2009 2:40:56 PM org.hp.openvms.lucene.index.FileIndexerJSP
>>> >> >> jspMain
>>> >> >> SEVERE: Indexing not completebackground merge hit exception:
>>> >> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [
>>> >> >> optimize]
>>> >> >> java.io.IOException: background merge hit exception: _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 into _b5 [optimize
>>> >> >> ]
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2273)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2218)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2198)
>>> >> >>        at
>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
>>> >> >> Source)
>>> >> >>        at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
>>> Source)
>>> >> >> Caused by: java.io.FileNotFoundException:
>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_61.cfs (i/o error (errno:5))
>>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>>> >> >>        at
>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
>>> >> >>        at
>>> >> >> org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.CompoundFileReader.<init>(CompoundFileReader.java:70)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.SegmentReader.initialize(SegmentReader.java:321)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:306)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.SegmentReader.get(SegmentReader.java:260)
>>> >> >>        at
>>> >> >>
>>> org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4220)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3884)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:205)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:260)
>>> >> >> IW 0 [main]: now flush at close
>>> >> >> IW 0 [main]:   flush: segment=null docStoreSegment=null
>>> >> docStoreOffset=0
>>> >> >> flushDocs=false flushDeletes=true flushDocStores=false numD
>>> >> >> ocs=0 numBufDelTerms=0
>>> >> >> IW 0 [main]:   index before flush _37:c465->_0 _4c:c165->_0
>>> >> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: CMS: now merge
>>> >> >> IW 0 [main]: CMS:   index: _37:c465->_0 _4c:c165->_0 _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0
>>> >> >> IW 0 [main]: CMS:   no more merges pending; now return
>>> >> >> IW 0 [main]: now call final commit()
>>> >> >> IW 0 [main]: startCommit(): start sizeInBytes=0
>>> >> >> IW 0 [main]: startCommit index=_37:c465->_0 _4c:c165->_0
>>> _61:c245->_0
>>> >> >> _9e:c485->_0 _ay:c225->_0 _az:c5->_0 _b0:c5->_0 changeCount=11
>>> >> >> 95
>>> >> >> IW 0 [main]: now sync _37.cfs
>>> >> >> IW 0 [main]: hit exception while closing
>>> >> >> Exception in thread "main" java.io.FileNotFoundException:
>>> >> >> /SYS$SYSDEVICE/RISHI/melon_1600/_37.cfs (i/o error (errno:5))
>>> >> >>        at java.io.RandomAccessFile.open(Native Method)
>>> >> >>        at
>>> java.io.RandomAccessFile.<init>(RandomAccessFile.java:231)
>>> >> >>        at
>>> >> org.apache.lucene.store.FSDirectory.sync(FSDirectory.java:456)
>>> >> >>        at
>>> >> >>
>>> org.apache.lucene.index.IndexWriter.startCommit(IndexWriter.java:4590)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:3369)
>>> >> >>        at
>>> >> >>
>>> >>
>>> org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:1654)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1602)
>>> >> >>        at
>>> >> >> org.apache.lucene.index.IndexWriter.close(IndexWriter.java:1578)
>>> >> >>        at
>>> org.hp.openvms.lucene.index.FileIndexerJSP.jspMain(Unknown
>>> >> >> Source)
>>> >> >>        at org.hp.openvms.lucene.index.FileIndexer.main(Unknown
>>> Source)
>>> >> >>
>>> >> >>
>>> >> >>
>>> >> >> I have setMaxBufferedDocs = 5 and setMergeFactor=5. I was trying
>>> >> >> different
>>> >> >> combinations of these to see if that helps but was unsuccessful.
>>> >> >>
>>> >> >>
>>> >> >> Regards,
>>> >> >> Rishi
>>> >> >> --
>>> >> >> View this message in context:
>>> >> >>
>>> >>
>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24951667.html
>>> >> >> Sent from the Lucene - Java Users mailing list archive at
>>> Nabble.com.
>>> >> >>
>>> >> >>
>>> >> >>
>>> ---------------------------------------------------------------------
>>> >> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>> >> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>> >> >>
>>> >> >>
>>> >> >
>>> >> >
>>> ---------------------------------------------------------------------
>>> >> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>> >> > For additional commands, e-mail: java-user-h...@lucene.apache.org
>>> >> >
>>> >> >
>>> >> >
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24952295.html
>>> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>> >>
>>> >>
>>> >> ---------------------------------------------------------------------
>>> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>> >>
>>> >>
>>> >
>>> >
>>>
>>> --
>>> View this message in context:
>>> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953575.html
>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: java-user-h...@lucene.apache.org
>>>
>>>
>>
>>
>
> --
> View this message in context: 
> http://www.nabble.com/Indexer-crashes-with-%22hit-exception-during-merge%22-tp24951667p24953833.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
For additional commands, e-mail: java-user-h...@lucene.apache.org

Reply via email to