>From this test, I would expect all 3 files to be left, because IndexWriter never gets another chance to remove the files.
IndexWriter only attempts to remove unreferenced files in roughly 3 places: * On open * On flushing a new segment * On finishing a merge So, the moment your optimize finished, IndexWriter wanted to delete 0.cfs and 1.cfs, but could not, because the currently open reader was using them. Then, because you don't close/open a new IndexWriter, no further attempt is made to delete. In a normal app, as you continue adding docs, normal merges kick off, etc, these unreferenced files would eventually be removed. It's just taking longer than it should, because IndexWriter doesn't know exactly when the reader was closed. I think we should add an explicit IndexWriter.removeUnreferencedFiles() method, so apps can control the immediacy if they want to. I'll open an issue. For your tests, if you add code to stop both threads, close the writer, then open & close a new writer, you should see 0.cfs and 1.cfs removed. Mike On Wed, Feb 10, 2010 at 9:54 AM, luocanrao <luocan19826...@sohu.com> wrote: > here a small test case I watched there are three compound files. > 0.cfs 6786kb > 1.cfs 2044kb > 2.cfs 8790kb(the optimize file) > I think in this testcase only 2.cfs left(the optimize file left),Is that > right?? > > import java.io.File; > import java.io.IOException; > > import org.apache.lucene.analysis.standard.StandardAnalyzer; > import org.apache.lucene.document.Document; > import org.apache.lucene.document.Field; > import org.apache.lucene.document.Fieldable; > import org.apache.lucene.index.CorruptIndexException; > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.IndexWriter; > import org.apache.lucene.store.FSDirectory; > import org.apache.lucene.store.LockObtainFailedException; > import org.apache.lucene.util.Version; > > public class ReopenTest { > IndexWriter writer; > > public static void main(String[] args) throws CorruptIndexException, > LockObtainFailedException, IOException { > final IndexWriter writer = new > IndexWriter(FSDirectory.open(new File( > "data")), new > StandardAnalyzer(Version.LUCENE_CURRENT), > IndexWriter.MaxFieldLength.LIMITED); > writer.setMergeFactor(16); > > Thread writeThread = new Thread(new Runnable() { > @Override > public void run() { > // write test data > try { > for (int i = 0; i < 1000000; i++) { > Document doc = new Document(); > Fieldable itemID = new > Field("ItemID", String > .valueOf(i), > Field.Store.YES, > > Field.Index.ANALYZED); > > writer.addDocument(doc); > } > > Thread.currentThread().sleep(10000); > System.out.println("optimize begin"); > writer.optimize(); > System.out.println("optimize end"); > } catch (CorruptIndexException e) { > e.printStackTrace(); > } catch (IOException e) { > e.printStackTrace(); > } catch (InterruptedException e) { > e.printStackTrace(); > } > > } > }); > > writeThread.start(); > > > Thread reopenThread = new Thread(new Runnable() { > public void run() { > try { > IndexReader reader = > writer.getReader(); > while (true) { > > Thread.currentThread().sleep(1000); > IndexReader newReader = > writer.getReader(); > if (newReader != reader) { > reader.close(); > reader = newReader; > } > System.out.println("132"); > } > > } catch (InterruptedException e) { > e.printStackTrace(); > } catch (IOException e) { > e.printStackTrace(); > } > } > > }); > > reopenThread.start(); > } > > } > > -----邮件原件----- > 发件人: luocan19826...@sohu.com [mailto:luocan19826...@sohu.com] > 发送时间: 2010年2月10日 22:49 > 收件人: java-user > 主题: RE: here a small test case problem:lucene did not delete old index file > after optimize method called > > I watched there are three compound files. > 0.cfs 6786kb > 1.cfs 2044kb > 2.cfs 8790kb(the optimize file) > I think in this testcase only 2.cfs left(the optimize file left),Is that > right?? > <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" > /><o:p> </o:p> > import java.io.File; > import java.io.IOException; > <o:p> </o:p> > import org.apache.lucene.analysis.standard.StandardAnalyzer; > import org.apache.lucene.document.Document; > import org.apache.lucene.document.Field; import > org.apache.lucene.document.Fieldable; > import org.apache.lucene.index.CorruptIndexException; > import org.apache.lucene.index.IndexReader; > import org.apache.lucene.index.IndexWriter; > import org.apache.lucene.store.FSDirectory; > import org.apache.lucene.store.LockObtainFailedException; > import org.apache.lucene.util.Version; > <o:p> </o:p> > public class ReopenTest { > IndexWriter writer; > <o:p> </o:p> > public static void > main(String[] args) throws CorruptIndexException, > > LockObtainFailedException, IOException { > > final IndexWriter writer = new > IndexWriter(FSDirectory.open(new File( > > "data")), new > StandardAnalyzer(Version.LUCENE_CURRENT), > > IndexWriter.MaxFieldLength.LIMITED); > > writer.setMergeFactor(16); > <o:p> </o:p> > > Thread writeThread = new Thread(new Runnable() { > > @Override > > public void run() { > > // write test data > > try { > > for (int i = 0; i < 1000000; i++) { > > Document doc = new > Document(); > > Fieldable itemID = new > Field("ItemID", String > > .valueOf(i), > Field.Store.YES, > > Field.Index. > ANALYZED); > <o:p> </o:p> > > writer.addDocument(doc); > > } > <o:p> </o:p> > > Thread.currentThread().sleep(10000); > > System.out.println("optimize > begin"); > > writer.optimize(); > > System.out.println("optimize end"); > > } catch (CorruptIndexException e) { > > e.printStackTrace(); > > } catch (IOException e) { > > e.printStackTrace(); > > } catch (InterruptedException e) { > > e.printStackTrace(); > > } > <o:p> </o:p> > > } > > }); > <o:p> </o:p> > > writeThread.start(); > > > > Thread reopenThread = new Thread(new Runnable() { > > public void run() { > > try { > > IndexReader reader = > writer.getReader(); > > while (true) { > > Thread.currentThread().sleep(1000); > > IndexReader newReader = > writer.getReader(); > > if (newReader != reader) { > > reader.close(); > > reader = newReader; > > } > > System.out.println("132"); > > } > <o:p> </o:p> > > } catch (InterruptedException e) { > > e.printStackTrace(); > > } catch (IOException e) { > > e.printStackTrace(); > > } > > } > <o:p> </o:p> > > }); > > > reopenThread.start(); > } > <o:p> </o:p> > } > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org