Re: Index corruption and repair

2022-05-25 Thread Antony Joseph
Hi Mike, Any updates? Regards, Antony On Wed, 11 May 2022 at 01:02, Antony Joseph wrote: > Hello Mike, > > 1. As requested, the full checkindex log is attached. > > 2. We haven't made any changes to the IndexDeletionPolicy - so the > assumption is the default policy is being used. > > 3. No,

Re: Index corruption and repair

2022-05-10 Thread Antony Joseph
Hello Mike, 1. As requested, the full checkindex log is attached. 2. We haven't made any changes to the IndexDeletionPolicy - so the assumption is the default policy is being used. 3. No, we are not using near-real-time readers. We are using filesystem based readers only (passing Directory to

Re: Index corruption and repair

2022-05-05 Thread Michael McCandless
Antony, do you maybe have Microsoft Defender turned on, which might quarantine files that it suspects are malicious? I'm not sure if it is on by default these days on modern Windows boxes ... Mike McCandless http://blog.mikemccandless.com On Thu, May 5, 2022 at 10:34 AM Michael McCandless <

Re: Index corruption and repair

2022-05-05 Thread Michael McCandless
On Thu, May 5, 2022 at 10:30 AM Uwe Schindler wrote: To find all errors in an index, you should pass -ea to the java command > line to enable assertions. > +1 Tempting to make CheckIndex demand that :) Or at least, slow you down and make it clear why, if assertions are disabled. Mike

Re: Index corruption and repair

2022-05-05 Thread Uwe Schindler
Hi, To find all errors in an index, you should pass -ea to the java command line to enable assertions. Uwe Am 5. Mai 2022 14:25:03 UTC schrieb Michael McCandless : >Hi Antony, > >Sorry for the late reply. > >Indeed the file _14gb.si is missing, yet _14gb.cfs is present (interesting >-- must

Re: Index corruption and repair

2022-05-05 Thread Michael McCandless
Hi Antony, Sorry for the late reply. Indeed the file _14gb.si is missing, yet _14gb.cfs is present (interesting -- must have failed deletion because an IndexReader has it open). And yet when you run CheckIndex on this directory (without -exorcise), the index is fine? No errors reported? Can

Re: Index corruption and repair

2022-05-04 Thread Antony Joseph
Hi Michael, Any update? Regards, Antony On Sun, 1 May 2022 at 19:35, Antony Joseph wrote: > Hi Michael, > > Thank you for your reply. Please find responses to your questions below. > > Regards, > Antony > > On Sat, 30 Apr 2022 at 18:59, Michael McCandless < > luc...@mikemccandless.com> wrote:

Re: Index corruption and repair

2022-05-01 Thread Antony Joseph
Hi Michael, Thank you for your reply. Please find responses to your questions below. Regards, Antony On Sat, 30 Apr 2022 at 18:59, Michael McCandless wrote: > Hi Antony, > > Hmm it looks like the root cause is this: > > Caused by: java.nio.file.NoSuchFileException: D:\i\202204\_14gb.si

Re: Index corruption and repair

2022-04-30 Thread Michael McCandless
Hi Antony, Hmm it looks like the root cause is this: Caused by: java.nio.file.NoSuchFileException: D:\i\202204\_14gb.si Can you list all the files in the index directory at the time this exception happens, and reply here? We need to figure out whether the file is really missing or what.

Re: Index corruption and repair

2022-04-30 Thread Antony Joseph
Thank you for your reply. *The full stack trace is included:* , > Java stacktrace: org.apache.lucene.index.CorruptIndexException: Unexpected file read error while reading index. (resource=BufferedChecksumIndexInput(MMapIndexInput(path="D:\i\202204\segments_10fj"))) at

Re: Index corruption and repair

2022-04-29 Thread Robert Muir
The most helpful thing would be the full stacktrace of the exception. This exception should be chaining the original exception and call site, and maybe tell us more about this error you hit. To me, it looks like a windows-specific issue where the filesystem is returning an unexpected error. So it

Re: Index corruption and repair

2022-04-28 Thread Antony Joseph
Thank you for your reply. This isn't happening in a single environment. Our application is being used by various clients and this has been reported by multiple users - all of whom were running the earlier pylucene (v4.10) - without issues. One thing to mention is that our earlier version used

Re: Index corruption and repair

2022-04-28 Thread Adrien Grand
Hi Anthony, This isn't something that you should try to fix programmatically, corruptions indicate that something is wrong with the environment, like a broken disk or corrupt RAM. I would suggest running a memtest to check your RAM and looking at system logs in case they have anything to tell

Index corruption and repair

2022-04-28 Thread Antony Joseph
Hello, We are facing a strange situation in our application as described below: *Using*: - Python 3.8.10 - Pylucene 6.5.0 - Java 8 (1.8.0_181) - Runs on Linux and Windows (error seen on Windows) We suddenly get the following *error*: 2022-02-10 09:58:09.253215: ERROR : writer |