Hi Glen:
I think it is in your application code:
The indexReader returned is not closed if the underlying index has changed.
If your update rate is high, you will run into this issue because GC may not
have caught up with the FH leak.
THe code should instead be:
if (indexReader!=null){
IndexReader newReader = indexReader.reopen();
if (newReader!=indexReader){
indexReader.close(); // close the old reader
}
}
However, with Lucene 2.9, you can also use:
IndexReader r = writer.getReader();, which is a much simpler API. You still
need to close the reader when you are done with them however.
-John
On Sun, Oct 18, 2009 at 10:47 AM, GlenAbbeyDrive <wende.l...@gmail.com>wrote:
>
> I commit the IndexWriter every 200 documents in a batch as follows and you
> can see that I reopened the reader after the commit.
>
> private void commit(IndexWriter writer) throws CorruptIndexException,
> IOException, SQLException {
> writer.commit();
> if(indexReader!=null)
> indexReader.reopen();
> }
>
> What do you mean by real-time reader?
>
> Thanks.
>
>
> Michael McCandless-2 wrote:
> >
> > Hmm, not good. Can you share more details about how your app is using
> > Lucene?
> >
> > Do you also have an IndexReader that's open on this directory? Do you
> > reopen it after indexing documents? If so, using IndexReader.reopen
> > or by closing and opening a new IndexReader? Or, by getting a near
> > real-time reader?
> >
> > Are you closing the IndexWriter after each batch, or calling commit?
> >
> > Mike
> >
> > On Sun, Oct 18, 2009 at 11:14 AM, GlenAbbeyDrive <wende.l...@gmail.com>
> > wrote:
> >>
> >> I switched from Lucene 2.4.0 to the latest 2.9.0 version and got too
> many
> >> files open within a few hours from my indexing process. Our indexing
> >> Java
> >> process adds about 2000 documents/minute.
> >> The IndexWriter (iw) has the following settings:
> >>
> >> iw.setMaxFieldLength(1024*1024*1024); // 1G as maximum terms.
> >> iw.setMergeFactor(10);
> >> iw.setMaxMergeDocs(100000);
> >> iw.setMaxBufferedDocs(1000);
> >> iw.setRAMBufferSizeMB(128);
> >> iw.setUseCompoundFile(true);
> >>
> >>
> >>
> >> The 'lsof' result shows rapidly growing number of open files in
> 'deleted'
> >> state. Counting the number of these:
> >>
> >> [server-1]#date
> >> Sun Oct 18 11:02:37 EDT 2009
> >> [server-1]# lsof -p 19606 |grep deleted |wc -l
> >> 3187
> >> [server-1]# date
> >> Sun Oct 18 11:07:29 EDT 2009
> >> [server-1]# lsof -p 19606 |grep deleted |wc -l
> >> 3912
> >>
> >> Here is a sample of those open files left behind in 'deleted' state:
> >>
> >> java 19606 root 1861r REG 8,17 273561 368673054
> >> /archive/index/_5fq.cfs (deleted)
> >> java 19606 root 1862r REG 8,17 9428 368673053
> >> /archive/index/_5fp.cfs (deleted)
> >> java 19606 root 1863r REG 8,17 363690 368673052
> >> /archive/index/_5fo.cfs (deleted)
> >> java 19606 root 1864r REG 8,17 47732 368673051
> >> /archive/index/_5fn.cfs (deleted)
> >> java 19606 root 1865r REG 8,17 417453 368673050
> >> /archive/index/_5fm.cfs (deleted)
> >> java 19606 root 1866r REG 8,17 1518126 368673049
> >> /archive/index/_5fl.cfs (deleted)
> >> java 19606 root 1871r REG 8,17 1236023 368673057
> >> /archive/index/_5ft.cfs (deleted)
> >> java 19606 root 1872r REG 8,17 29743 368673055
> >> /archive/index/_5fr.cfs (deleted)
> >> java 19606 root 1873r REG 8,17 273561 368673054
> >> /archive/index/_5fq.cfs (deleted)
> >> java 19606 root 1874r REG 8,17 9428 368673053
> >> /archive/index/_5fp.cfs (deleted)
> >> java 19606 root 1875r REG 8,17 363690 368673052
> >> /archive/index/_5fo.cfs (deleted)
> >> java 19606 root 1876r REG 8,17 47732 368673051
> >> /archive/index/_5fn.cfs (deleted)
> >> java 19606 root 1877r REG 8,17 417453 368673050
> >> /archive/index/_5fm.cfs (deleted)
> >> java 19606 root 1878r REG 8,17 1518126 368673049
> >> /archive/index/_5fl.cfs (deleted)
> >> java 19606 root 1883r REG 8,17 1389856 368673056
> >> /archive/index/_5fs.cfs (deleted)
> >> java 19606 root 1884r REG 8,17 29743 368673055
> >> /archive/index/_5fr.cfs (deleted)
> >> java 19606 root 1885r REG 8,17 273561 368673054
> >> /archive/index/_5fq.cfs (deleted)
> >> java 19606 root 1886r REG 8,17 9428 368673053
> >> /archive/index/_5fp.cfs (deleted)
> >> java 19606 root 1887r REG 8,17 363690 368673052
> >> /archive/index/_5fo.cfs (deleted)
> >> java 19606 root 1888r REG 8,17 47732 368673051
> >> /archive/index/_5fn.cfs (deleted)
> >> java 19606 root 1889r REG 8,17 417453 368673050
> >> /archive/index/_5fm.cfs (deleted)
> >> java 19606 root 1890r REG 8,17 1518126 368673049
> >> /archive/index/_5fl.cfs (deleted)
> >> java 19606 root 1895r REG 8,17 1389856 368673056
> >> /archive/index/_5fs.cfs (deleted)
> >> java 19606 root 1896r REG 8,17 29743 368673055
> >> /archive/index/_5fr.cfs (deleted)
> >> java 19606 root 1897r REG 8,17 273561 368673054
> >> /archive/index/_5fq.cfs (deleted)
> >> java 19606 root 1898r REG 8,17 9428 368673053
> >> /archive/index/_5fp.cfs (deleted)
> >> java 19606 root 1899r REG 8,17 363690 368673052
> >> /archive/index/_5fo.cfs (deleted)
> >> java 19606 root 1900r REG 8,17 47732 368673051
> >> /archive/index/_5fn.cfs (deleted)
> >> java 19606 root 1901r REG 8,17 417453 368673050
> >> /archive/index/_5fm.cfs (deleted)
> >> java 19606 root 1902r REG 8,17 1518126 368673049
> >> /archive/index/_5fl.cfs (deleted)
> >> java 19606 root 1910r REG 8,17 403610 368673058
> >> /archive/index/_5fu.cfs (deleted)
> >> java 19606 root 1911r REG 8,17 1236023 368673057
> >> /archive/index/_5ft.cfs (deleted)
> >> java 19606 root 1912r REG 8,17 1389856 368673056
> >> /archive/index/_5fs.cfs (deleted)
> >> java 19606 root 1913r REG 8,17 29743 368673055
> >> /archive/index/_5fr.cfs (deleted)
> >> java 19606 root 1914r REG 8,17 273561 368673054
> >> /archive/index/_5fq.cfs (deleted)
> >> java 19606 root 1915r REG 8,17 9428 368673053
> >> /archive/index/_5fp.cfs (deleted)
> >> java 19606 root 1916r REG 8,17 363690 368673052
> >> /archive/index/_5fo.cfs (deleted)
> >> java 19606 root 1917r REG 8,17 47732 368673051
> >> /archive/index/_5fn.cfs (deleted)
> >> java 19606 root 1918r REG 8,17 417453 368673050
> >> /archive/index/_5fm.cfs (deleted)
> >> java 19606 root 1919r REG 8,17 1518126 368673049
> >> /archive/index/_5fl.cfs (deleted)
> >>
> >>
> >> --
> >> View this message in context:
> >>
> http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25947462.html
> >> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> >> For additional commands, e-mail: java-user-h...@lucene.apache.org
> >>
> >>
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: java-user-h...@lucene.apache.org
> >
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25948812.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org
> For additional commands, e-mail: java-user-h...@lucene.apache.org
>
>
