Sorry about the previous stack trace -- that was a wild goose-chase. Here is the real culprit:
at org.apache.lucene.index.SegmentTermEnum.growBuffer(SegmentTermEnum.java(Compiled Code)) at org.apache.lucene.index.SegmentTermEnum.readTerm(SegmentTermEnum.java(Compiled Code)) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java(Compiled Code)) at org.apache.lucene.index.TermInfosReader.scanEnum(TermInfosReader.java(Compiled Code)) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java(Compiled Code)) at org.apache.lucene.index.SegmentReader.docFreq(SegmentReader.java(Compiled Code)) at org.apache.lucene.index.MultiReader.docFreq(MultiReader.java(Compiled Code)) at org.apache.lucene.search.IndexSearcher.docFreq(IndexSearcher.java:69) at org.apache.lucene.search.Similarity.idf(Similarity.java:255) at org.apache.lucene.search.TermQuery$TermWeight.sumOfSquaredWeights(TermQuery.java:47) at org.apache.lucene.search.BooleanQuery$BooleanWeight.sumOfSquaredWeights(BooleanQuery.java:110) at org.apache.lucene.search.BooleanQuery$BooleanWeight.sumOfSquaredWeights(BooleanQuery.java:110) at org.apache.lucene.search.Query.weight(Query.java(Compiled Code)) at org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java(Compiled Code)) at org.apache.lucene.search.Hits.getMoreDocs(Hits.java(Compiled Code)) at org.apache.lucene.search.Hits.<init>(Hits.java:43) at org.apache.lucene.search.Searcher.search(Searcher.java(Compiled Code)) the growBuffer contains: buffer = new char[length] length is the result of two readVInt() on the "tii" file (start + length) I made a local copy of the largest "tii" file and (using beanshell) read the two "variable ints": bsh % d = org.apache.lucene.store.FSDirectory.getDirectory("/tmp/", false); bsh % i = d.openFile("copy.tii"); bsh % print (i.readVInt()); 266338303 bsh % print (i.readVInt()); 0 In other words (If I read the code correctly) _start_ is set to 266338303 while 'length' is set to 0. growBuffer allocates a char array, and (according to our GC logs) allocates 532676624 bytes (which is 266338303*2 + 18) I am guessing the 18 additional bytes is type information, which is probably JVM-dependent. So now the question becomes more direct: Why on earth does our biggest segment think that it starts at 266 million characters? I just noticed that the hex value of the 266-million-tingy is ... ... drumroll ... ... fdfffff Now I'm thinking that either the read/write-operations dont work 100% or that there's a corrupt index. However we _have_ tried to rebuild the index from scratch, so I don't think it's a corrupt index. And finally, the start of the 'tii' file is as follows: $ hexdump /tmp/test.tii | head 0000000 ffff feff 0000 0000 0000 f3cc 0000 8000 0000010 0000 1000 0000 0000 0000 0014 3107 3030 0000020 3130 3831 0104 02fd 897f 0407 3203 3034 0000030 0104 0380 0180 0788 0304 3633 0430 8001 Regards, Fredrik ------------------------------------------------------------------------- Start.no tilbyr nå raskere bredbånd til lavere pris. Sjekk http://www.start.no/bredband/ for mer informasjon --------------------------------------------------------------------- To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]