Re: Corrupt segments file full of zeros

2011-06-29 Thread Michael McCandless
On Tue, Jun 28, 2011 at 10:45 PM, Trejkaz wrote: > On Wed, Jun 29, 2011 at 2:24 AM, Michael McCandless > wrote: >> Here's the issue: >> >>    https://issues.apache.org/jira/browse/LUCENE-3255 >> >> It's because we read the first 0 int to be an ancient segments file >> format, and the next 0 int t

Re: Corrupt segments file full of zeros

2011-06-28 Thread Trejkaz
On Wed, Jun 29, 2011 at 2:24 AM, Michael McCandless wrote: > Here's the issue: > >    https://issues.apache.org/jira/browse/LUCENE-3255 > > It's because we read the first 0 int to be an ancient segments file > format, and the next 0 int to mean there are no segments.  Yuck! > > This format pre-dat

Re: Corrupt segments file full of zeros

2011-06-28 Thread Michael McCandless
age > From: Michael McCandless > To: java-user@lucene.apache.org > Sent: Tue, 28 June, 2011 14:59:48 > Subject: Re: Corrupt segments file full of zeros > > On Tue, Jun 28, 2011 at 9:29 AM, mark harwood wrote: >> Hi Mike. >>>>Hmmm -- what code are you running here, to pr

Re: Corrupt segments file full of zeros

2011-06-28 Thread mark harwood
From: Michael McCandless To: java-user@lucene.apache.org Sent: Tue, 28 June, 2011 14:59:48 Subject: Re: Corrupt segments file full of zeros On Tue, Jun 28, 2011 at 9:29 AM, mark harwood wrote: > Hi Mike. >>>Hmmm -- what code are you running here, to pr

Re: Corrupt segments file full of zeros

2011-06-28 Thread Michael McCandless
On Tue, Jun 28, 2011 at 9:29 AM, mark harwood wrote: > Hi Mike. >>>Hmmm -- what code are you running here, to print the number of docs? > > SegmentInfos.setInfoStream(System.out); > FSDirectory dir = FSDirectory.open(new File("j:/indexes/myindex")); > IndexReader r = IndexReader.open(dir, true); >

Re: Corrupt segments file full of zeros

2011-06-28 Thread mark harwood
e way the above test correctly sees _3 as corrupt. Cheers Mark - Original Message From: Michael McCandless To: java-user@lucene.apache.org Sent: Tue, 28 June, 2011 14:04:40 Subject: Re: Corrupt segments file full of zeros On Tue, Jun 28, 2011 at 8:53 AM, Tarr, Gregory wrote: > M

Re: Corrupt segments file full of zeros

2011-06-28 Thread Michael McCandless
On Tue, Jun 28, 2011 at 8:53 AM, Tarr, Gregory wrote: > Michael > > We are not using commit points unfortunately. That's fine -- even if you don't keep multiple commit points in your index, when a commit() op fails, then you can end up with two segments_N files. The older one is "good" (last suc

RE: Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
. My question is: why does an index with 4vc open? Thanks Greg -Original Message- From: Michael McCandless [mailto:luc...@mikemccandless.com] Sent: 28 June 2011 13:36 To: java-user@lucene.apache.org Subject: Re: Corrupt segments file full of zeros OK, this is why Lucene (and Luke

Re: Corrupt segments file full of zeros

2011-06-28 Thread Michael McCandless
the index from being wiped. > > Thanks > > Greg > > -Original Message- > From: Michael McCandless [mailto:luc...@mikemccandless.com] > Sent: 28 June 2011 13:26 > To: java-user@lucene.apache.org > Subject: Re: Corrupt segments file full of zeros > > Is t

RE: Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
...@mikemccandless.com] Sent: 28 June 2011 13:26 To: java-user@lucene.apache.org Subject: Re: Corrupt segments file full of zeros Is there only one segments_N file in the index (the one with all 0s)? Or is there a segments_(N-1) too? Mike McCandless http://blog.mikemccandless.com On Tue, Jun 28, 2011 at 8:17

Re: Corrupt segments file full of zeros

2011-06-28 Thread Michael McCandless
dler > To: java-user@lucene.apache.org > Sent: Tue, 28 June, 2011 12:32:34 > Subject: RE: Corrupt segments file full of zeros > > So where is the problem at all? Why should a segments file not contain > lots > of zeroes? If the index is not corrupt all is fine. >

RE: Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
June 2011 13:09 To: java-user@lucene.apache.org Subject: Re: Corrupt segments file full of zeros According to the spec there should at least be an Int32 of -9 to declare the Format - http://lucene.apache.org/java/2_9_3/fileformats.html#Segments File - Original Message From: Uwe Schi

Re: Corrupt segments file full of zeros

2011-06-28 Thread mark harwood
; From: Tarr, Gregory [mailto:gregory.t...@detica.com] > Sent: Tuesday, June 28, 2011 11:56 AM > To: java-user@lucene.apache.org > Subject: RE: Corrupt segments file full of zeros > > Yes I have done that, and you just get "No problems were detected with this > index" > &

RE: Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
age- From: Uwe Schindler [mailto:u...@thetaphi.de] Sent: 28 June 2011 12:33 To: java-user@lucene.apache.org Subject: RE: Corrupt segments file full of zeros So where is the problem at all? Why should a segments file not contain lots of zeroes? If the index is not corrupt all is fine. -

RE: Corrupt segments file full of zeros

2011-06-28 Thread Uwe Schindler
regory [mailto:gregory.t...@detica.com] > Sent: Tuesday, June 28, 2011 11:56 AM > To: java-user@lucene.apache.org > Subject: RE: Corrupt segments file full of zeros > > Yes I have done that, and you just get "No problems were detected with this > index" > > Surely there is a

RE: Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
- From: Shai Erera [mailto:ser...@gmail.com] Sent: 28 June 2011 10:36 To: java-user@lucene.apache.org Subject: Re: Corrupt segments file full of zeros You can try the CheckIndex tool. You feed it a directory and call .check() and it reports the results. Shai On Tue, Jun 28, 2011 at 11:4

Re: Corrupt segments file full of zeros

2011-06-28 Thread Shai Erera
You can try the CheckIndex tool. You feed it a directory and call .check() and it reports the results. Shai On Tue, Jun 28, 2011 at 11:46 AM, Tarr, Gregory wrote: > We have a problem with our fileserver where our indexes are hosted > remotely, using Lucene 2.9.3. > > This can mean that a segment

Corrupt segments file full of zeros

2011-06-28 Thread Tarr, Gregory
We have a problem with our fileserver where our indexes are hosted remotely, using Lucene 2.9.3. This can mean that a segments file is written which is full of ASCII zeros. Using the od -ah command, we get: 000 nul nul nul nul nul nul nuletc If opened in Luke, the index opens successfull