Hi We scan web and index pages in lucene. Our index size is in the range of 500K to 1 million documens. As we index pages, we also call IndexWriter.optimize after certain time intervals [I believe Lucene also does optimization in the background ?]. So far it has worked great. But for just this one scan we noticed that the our index size grew to 90 GB for about 900K documents [typical index size should be around 17-18GB]. We are not sure what caused the index to grow this large. Outside of our system, when we did a forced IndexWriter.optimize() on this 90 GB lucene index, it indeed shrinked to 17 GB. My question is what may have caused the size to grow to 90GB? Did the size grow because optimization failed ? Does optimization fail if there is any foreign file in the lucene index directory [though we tried optimizing with foreign files in lucene directory, and lucene still did optimize the index.]
any suggestion, input will be quite valuable. thanks Pratyush -- View this message in context: http://www.nabble.com/InderxWriter.optimize%28%29-fail-tp21937277p21937277.html Sent from the Lucene - General mailing list archive at Nabble.com.
