Let me explain the usage scenario: I have an application with frequent doucment updates and it uses NRT reader. I also have a background thread which calls commit on the IndexWriter on a regular interval. There is one instance of IndexWriter which is created and it is never closed.
The problem is that Lucene is creating lot of files and huge index which is way much more than the size of documents. This size is not reduced despite the lucene merge running in the background. (I am using the default TieredMergePolicy and ConcurrentMergeScheduler). I also tried running a background thread calling IndexWriter.maybeMerge() at regular interval still the size of the index is not changing. I then tried calling IndexWriter.close() at regular interval and found that this actually reduces the size of index and performs merging. So my question is what is the suggested way of using IndexWriter? Do I need to close it on regular intervals? Also why merge which is running in the background is not reducing the index size? Thanks, Anand Bhagwat.
