Thanks, just wanted to make one other comment. In my case, there were no "segments_..." files created prior to the crash, so I don't think we can safely rely on their presence/content.
On Thu, May 30, 2013 at 8:21 AM, Michael McCandless < luc...@mikemccandless.com> wrote: > I opened https://issues.apache.org/jira/browse/LUCENE-5024 to see if > we can improve this ... > > Mike McCandless > > http://blog.mikemccandless.com > > > On Thu, May 30, 2013 at 7:31 AM, Michael McCandless > <luc...@mikemccandless.com> wrote: > > Hi Vitaly, > > > > This is unfortunately due to a recent change in IndexWriter ... search > > for the recent thread "CorruptIndexException when opening Index during > > first commit" on this list. > > > > Normally, if a commit fails for any reason (OS, hardware, JVM crashes) > > we simply fall back to the last commit and no intervention is > > necessary. > > > > But if that commit was the very first commit ... we used to try to > > detect this and treat it as if there were no index present. This is > > the ideal/correct behavior, because "no index present" was in fact the > > state of the index before this failed commit. > > > > However that logic was dangerous and would sometimes result in > > IndexWriter thinking no index was present when in fact there was one, > > when there were transient errors e.g. due to file descriptor > > exhaustion. So to be safe we now throw the CorruptIndexException on > > first commit back to the application. > > > > Your workaround sounds good. Another option instead of the separate > > init file, on hitting a CorruptIndexException, might be to list the > > directory: if there is only one file, segments_1, then it's a corrupt > > first commit and you can recreate the index. > > > > Mike McCandless > > > > http://blog.mikemccandless.com > > > > On Wed, May 29, 2013 at 8:09 PM, Vitaly Funstein <vfunst...@gmail.com> > wrote: > >> I have encountered a strange issue, that appears to be pretty hard to > hit, > >> but is still a serious problem when it does occur. It seems that if the > JVM > >> crashes in a racy fashion with instantiation of IndexWriter, the index > may > >> be left in an inconsistent state. An attempt to reload such an index on > >> restart will result in a CorruptIndexException thrown when calling the > >> constructor again. > >> > >> My idea of a workaround involves the following: > >> > >> - call commit() immediately after invoking the constructor. > >> - create a special init file after the commit to indicate the index was > >> created successfully > >> - on app startup, check to see if the init file is there; if not, the > index > >> is corrupt and needs to be recreated. > >> > >> (The last two already happen in my code) > >> > >> Is this a valid approach to protect from this sort of race? Is there > >> already a built-in mechanism to make constructor fully synchronous wrt > >> persisting all the metadata? If not, it seems like there probably should > >> be... > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > >
