Which version of Lucene, and which OS/IO system? Is it possible fsync lies in your env?
Mike On Fri, Jul 9, 2010 at 7:59 AM, Earwin Burrfoot <[email protected]> wrote: > I believe I've seen a similar condition a few times. > A segments file referring zero-length segment files after a disk full event. > > On Fri, Jul 9, 2010 at 13:37, Michael McCandless > <[email protected]> wrote: >> I responded on the original thread. >> >> Disk full should never cause index corruption except on very old >> versions of Lucene... >> >> Mike >> >> On Thu, Jul 8, 2010 at 10:40 PM, Lance Norskog <[email protected]> wrote: >>> I think he saying that there is a race condition involving writing out >>> the buffered changes from ram, doing deletes, doing commits and >>> writing out new segment files when Lucene runs out of disk. This race >>> condition (or possibly a swallowed I/O Exception) may cause bogus >>> segment files, and the index becomes unuseable even though the >>> existing files make up a clean index. >>> >>> On Thu, Jul 8, 2010 at 5:29 PM, Lance Norskog <[email protected]> wrote: >>>> Forwarded to lucene-dev as a possible bug. >>>> >>>> On Wed, Jul 7, 2010 at 7:12 PM, Li Li <[email protected]> wrote: >>>>> I use SegmentInfos to read the segment_N file and found the error is >>>>> that it try to load deletedDocs but the .del file's size is 0(because >>>>> of disk error) . So I use SegmentInfos to set delGen=-1 to ignore >>>>> deleted Docs. >>>>> But I think there is some bug. The logic of write my be -- it first >>>>> writes the .del file then write the segment_N file. But it only write >>>>> to buffer and don't flush to disk immediately. So when disk full. it >>>>> may happen that segment_N file is flushed but del file faild. >>>>> >>>>> 2010/7/8 Lance Norskog <[email protected]>: >>>>>> If autocommit does not to an automatic rollback, that is a serious bug. >>>>>> >>>>>> There should be a way to detect that an automatic rollback has >>>>>> happened, but I don't know what it is. Maybe something in the Solr >>>>>> MBeans? >>>>>> >>>>>> On Wed, Jul 7, 2010 at 5:41 AM, osocurious2 <[email protected]> >>>>>> wrote: >>>>>>> >>>>>>> I haven't used this myself, but Solr supports a >>>>>>> http://wiki.apache.org/solr/UpdateXmlMessages#A.22rollback.22 rollback >>>>>>> function. It is supposed to rollback to the state at the previous >>>>>>> commit. So >>>>>>> you may want to turn off auto-commit on the index you are updating if >>>>>>> you >>>>>>> want to control what that last commit level is. >>>>>>> >>>>>>> However, in your case if the index gets corrupted due to a disk full >>>>>>> situation, I don't know what rollback would do, if anything, to help. >>>>>>> You >>>>>>> may need to play with the scenario to see what would happen. >>>>>>> >>>>>>> If you are using the DataImportHandler it may handle the rollback for >>>>>>> you...again, however, it may not deal with disk full situations >>>>>>> gracefully >>>>>>> either. >>>>>>> -- >>>>>>> View this message in context: >>>>>>> http://lucene.472066.n3.nabble.com/index-format-error-because-disk-full-tp948249p948968.html >>>>>>> Sent from the Solr - User mailing list archive at Nabble.com. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Lance Norskog >>>>>> [email protected] >>>>>> >>>>> >>>> >>>> >>>> >>>> -- >>>> Lance Norskog >>>> [email protected] >>>> >>> >>> >>> >>> -- >>> Lance Norskog >>> [email protected] >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: [email protected] >>> For additional commands, e-mail: [email protected] >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > > > -- > Kirill Zakharenko/Кирилл Захаренко ([email protected]) > Phone: +7 (495) 683-567-4 > ICQ: 104465785 > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > > --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
