Re: Slowing down (rate-limiting/throttling) IndexWriter.optimize

2008-08-23 Thread Michael McCandless
I think IO throttling would be a useful built-in feature. I imagine many people are actually uknowingly affected by this, when they make changes to their index on the same machine that also does simultaneous searching. It's not only optimize() that will cause this, but also normal flushin

Slowing down (rate-limiting/throttling) IndexWriter.optimize

2008-08-20 Thread Halsey, Stephen
Hi, We are using lucene to index a large number of documents (millions) and we currently optimize half the index in the background every 2 days, to stop it becoming too fragmented. This takes about an hour and we are finding during this time searches are slowed down dramatically on that machine.