OK I opened this issue: https://issues.apache.org/jira/browse/LUCENE-2259
And put a patch on. If you can try the patch, that'd be great :) You should be able to apply the patch, build a new jar, then run your test again unmodified, and 0.cfs, 1.cfs should then be removed. Mike 2010/2/10 Michael McCandless <luc...@mikemccandless.com>: > From this test, I would expect all 3 files to be left, because > IndexWriter never gets another chance to remove the files. > > IndexWriter only attempts to remove unreferenced files in roughly 3 places: > > * On open > * On flushing a new segment > * On finishing a merge > > So, the moment your optimize finished, IndexWriter wanted to delete > 0.cfs and 1.cfs, but could not, because the currently open reader was > using them. Then, because you don't close/open a new IndexWriter, no > further attempt is made to delete. > > In a normal app, as you continue adding docs, normal merges kick off, > etc, these unreferenced files would eventually be removed. > > It's just taking longer than it should, because IndexWriter doesn't > know exactly when the reader was closed. > > I think we should add an explicit > IndexWriter.removeUnreferencedFiles() method, so apps can control the > immediacy if they want to. I'll open an issue. > > For your tests, if you add code to stop both threads, close the > writer, then open & close a new writer, you should see 0.cfs and 1.cfs > removed. > > Mike > > On Wed, Feb 10, 2010 at 9:54 AM, luocanrao <luocan19826...@sohu.com> wrote: >> here a small test case I watched there are three compound files. >> 0.cfs 6786kb >> 1.cfs 2044kb >> 2.cfs 8790kb(the optimize file) >> I think in this testcase only 2.cfs left(the optimize file left),Is that >> right?? >> >> import java.io.File; >> import java.io.IOException; >> >> import org.apache.lucene.analysis.standard.StandardAnalyzer; >> import org.apache.lucene.document.Document; >> import org.apache.lucene.document.Field; >> import org.apache.lucene.document.Fieldable; >> import org.apache.lucene.index.CorruptIndexException; >> import org.apache.lucene.index.IndexReader; >> import org.apache.lucene.index.IndexWriter; >> import org.apache.lucene.store.FSDirectory; >> import org.apache.lucene.store.LockObtainFailedException; >> import org.apache.lucene.util.Version; >> >> public class ReopenTest { >> IndexWriter writer; >> >> public static void main(String[] args) throws CorruptIndexException, >> LockObtainFailedException, IOException { >> final IndexWriter writer = new >> IndexWriter(FSDirectory.open(new File( >> "data")), new >> StandardAnalyzer(Version.LUCENE_CURRENT), >> IndexWriter.MaxFieldLength.LIMITED); >> writer.setMergeFactor(16); >> >> Thread writeThread = new Thread(new Runnable() { >> @Override >> public void run() { >> // write test data >> try { >> for (int i = 0; i < 1000000; i++) { >> Document doc = new Document(); >> Fieldable itemID = new >> Field("ItemID", String >> .valueOf(i), >> Field.Store.YES, >> >> Field.Index.ANALYZED); >> >> writer.addDocument(doc); >> } >> >> Thread.currentThread().sleep(10000); >> System.out.println("optimize begin"); >> writer.optimize(); >> System.out.println("optimize end"); >> } catch (CorruptIndexException e) { >> e.printStackTrace(); >> } catch (IOException e) { >> e.printStackTrace(); >> } catch (InterruptedException e) { >> e.printStackTrace(); >> } >> >> } >> }); >> >> writeThread.start(); >> >> >> Thread reopenThread = new Thread(new Runnable() { >> public void run() { >> try { >> IndexReader reader = >> writer.getReader(); >> while (true) { >> >> Thread.currentThread().sleep(1000); >> IndexReader newReader = >> writer.getReader(); >> if (newReader != reader) { >> reader.close(); >> reader = newReader; >> } >> System.out.println("132"); >> } >> >> } catch (InterruptedException e) { >> e.printStackTrace(); >> } catch (IOException e) { >> e.printStackTrace(); >> } >> } >> >> }); >> >> reopenThread.start(); >> } >> >> } >> >> -----邮件原件----- >> 发件人: luocan19826...@sohu.com [mailto:luocan19826...@sohu.com] >> 发送时间: 2010年2月10日 22:49 >> 收件人: java-user >> 主题: RE: here a small test case problem:lucene did not delete old index file >> after optimize method called >> >> I watched there are three compound files. >> 0.cfs 6786kb >> 1.cfs 2044kb >> 2.cfs 8790kb(the optimize file) >> I think in this testcase only 2.cfs left(the optimize file left),Is that >> right?? >> <?xml:namespace prefix = o ns = "urn:schemas-microsoft-com:office:office" >> /><o:p> </o:p> >> import java.io.File; >> import java.io.IOException; >> <o:p> </o:p> >> import org.apache.lucene.analysis.standard.StandardAnalyzer; >> import org.apache.lucene.document.Document; >> import org.apache.lucene.document.Field; import >> org.apache.lucene.document.Fieldable; >> import org.apache.lucene.index.CorruptIndexException; >> import org.apache.lucene.index.IndexReader; >> import org.apache.lucene.index.IndexWriter; >> import org.apache.lucene.store.FSDirectory; >> import org.apache.lucene.store.LockObtainFailedException; >> import org.apache.lucene.util.Version; >> <o:p> </o:p> >> public class ReopenTest { >> IndexWriter writer; >> <o:p> </o:p> >> public static void >> main(String[] args) throws CorruptIndexException, >> >> LockObtainFailedException, IOException { >> >> final IndexWriter writer = new >> IndexWriter(FSDirectory.open(new File( >> >> "data")), new >> StandardAnalyzer(Version.LUCENE_CURRENT), >> >> IndexWriter.MaxFieldLength.LIMITED); >> >> writer.setMergeFactor(16); >> <o:p> </o:p> >> >> Thread writeThread = new Thread(new Runnable() { >> >> @Override >> >> public void run() { >> >> // write test data >> >> try { >> >> for (int i = 0; i < 1000000; i++) { >> >> Document doc = new >> Document(); >> >> Fieldable itemID = new >> Field("ItemID", String >> >> .valueOf(i), >> Field.Store.YES, >> >> Field.Index. >> ANALYZED); >> <o:p> </o:p> >> >> writer.addDocument(doc); >> >> } >> <o:p> </o:p> >> >> Thread.currentThread().sleep(10000); >> >> System.out.println("optimize >> begin"); >> >> writer.optimize(); >> >> System.out.println("optimize end"); >> >> } catch (CorruptIndexException e) { >> >> e.printStackTrace(); >> >> } catch (IOException e) { >> >> e.printStackTrace(); >> >> } catch (InterruptedException e) { >> >> e.printStackTrace(); >> >> } >> <o:p> </o:p> >> >> } >> >> }); >> <o:p> </o:p> >> >> writeThread.start(); >> >> >> >> Thread reopenThread = new Thread(new Runnable() { >> >> public void run() { >> >> try { >> >> IndexReader reader = >> writer.getReader(); >> >> while (true) { >> >> Thread.currentThread().sleep(1000); >> >> IndexReader newReader = >> writer.getReader(); >> >> if (newReader != reader) { >> >> reader.close(); >> >> reader = newReader; >> >> } >> >> System.out.println("132"); >> >> } >> <o:p> </o:p> >> >> } catch (InterruptedException e) { >> >> e.printStackTrace(); >> >> } catch (IOException e) { >> >> e.printStackTrace(); >> >> } >> >> } >> <o:p> </o:p> >> >> }); >> >> >> reopenThread.start(); >> } >> <o:p> </o:p> >> } >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org