Super! Thanks for bringing closure. Mike
On Sun, Oct 18, 2009 at 2:31 PM, GlenAbbeyDrive <wende.l...@gmail.com> wrote: > > Using a 'realtime' reader off the IndexWriter (writer.getReader()) instead of > indexReader.reopen() seems fixed this problem. > > > Michael McCandless-2 wrote: >> >> Hmm, not good. Can you share more details about how your app is using >> Lucene? >> >> Do you also have an IndexReader that's open on this directory? Do you >> reopen it after indexing documents? If so, using IndexReader.reopen >> or by closing and opening a new IndexReader? Or, by getting a near >> real-time reader? >> >> Are you closing the IndexWriter after each batch, or calling commit? >> >> Mike >> >> On Sun, Oct 18, 2009 at 11:14 AM, GlenAbbeyDrive <wende.l...@gmail.com> >> wrote: >>> >>> I switched from Lucene 2.4.0 to the latest 2.9.0 version and got too many >>> files open within a few hours from my indexing process. Our indexing >>> Java >>> process adds about 2000 documents/minute. >>> The IndexWriter (iw) has the following settings: >>> >>> iw.setMaxFieldLength(1024*1024*1024); // 1G as maximum terms. >>> iw.setMergeFactor(10); >>> iw.setMaxMergeDocs(100000); >>> iw.setMaxBufferedDocs(1000); >>> iw.setRAMBufferSizeMB(128); >>> iw.setUseCompoundFile(true); >>> >>> >>> >>> The 'lsof' result shows rapidly growing number of open files in 'deleted' >>> state. Counting the number of these: >>> >>> [server-1]#date >>> Sun Oct 18 11:02:37 EDT 2009 >>> [server-1]# lsof -p 19606 |grep deleted |wc -l >>> 3187 >>> [server-1]# date >>> Sun Oct 18 11:07:29 EDT 2009 >>> [server-1]# lsof -p 19606 |grep deleted |wc -l >>> 3912 >>> >>> Here is a sample of those open files left behind in 'deleted' state: >>> >>> java 19606 root 1861r REG 8,17 273561 368673054 >>> /archive/index/_5fq.cfs (deleted) >>> java 19606 root 1862r REG 8,17 9428 368673053 >>> /archive/index/_5fp.cfs (deleted) >>> java 19606 root 1863r REG 8,17 363690 368673052 >>> /archive/index/_5fo.cfs (deleted) >>> java 19606 root 1864r REG 8,17 47732 368673051 >>> /archive/index/_5fn.cfs (deleted) >>> java 19606 root 1865r REG 8,17 417453 368673050 >>> /archive/index/_5fm.cfs (deleted) >>> java 19606 root 1866r REG 8,17 1518126 368673049 >>> /archive/index/_5fl.cfs (deleted) >>> java 19606 root 1871r REG 8,17 1236023 368673057 >>> /archive/index/_5ft.cfs (deleted) >>> java 19606 root 1872r REG 8,17 29743 368673055 >>> /archive/index/_5fr.cfs (deleted) >>> java 19606 root 1873r REG 8,17 273561 368673054 >>> /archive/index/_5fq.cfs (deleted) >>> java 19606 root 1874r REG 8,17 9428 368673053 >>> /archive/index/_5fp.cfs (deleted) >>> java 19606 root 1875r REG 8,17 363690 368673052 >>> /archive/index/_5fo.cfs (deleted) >>> java 19606 root 1876r REG 8,17 47732 368673051 >>> /archive/index/_5fn.cfs (deleted) >>> java 19606 root 1877r REG 8,17 417453 368673050 >>> /archive/index/_5fm.cfs (deleted) >>> java 19606 root 1878r REG 8,17 1518126 368673049 >>> /archive/index/_5fl.cfs (deleted) >>> java 19606 root 1883r REG 8,17 1389856 368673056 >>> /archive/index/_5fs.cfs (deleted) >>> java 19606 root 1884r REG 8,17 29743 368673055 >>> /archive/index/_5fr.cfs (deleted) >>> java 19606 root 1885r REG 8,17 273561 368673054 >>> /archive/index/_5fq.cfs (deleted) >>> java 19606 root 1886r REG 8,17 9428 368673053 >>> /archive/index/_5fp.cfs (deleted) >>> java 19606 root 1887r REG 8,17 363690 368673052 >>> /archive/index/_5fo.cfs (deleted) >>> java 19606 root 1888r REG 8,17 47732 368673051 >>> /archive/index/_5fn.cfs (deleted) >>> java 19606 root 1889r REG 8,17 417453 368673050 >>> /archive/index/_5fm.cfs (deleted) >>> java 19606 root 1890r REG 8,17 1518126 368673049 >>> /archive/index/_5fl.cfs (deleted) >>> java 19606 root 1895r REG 8,17 1389856 368673056 >>> /archive/index/_5fs.cfs (deleted) >>> java 19606 root 1896r REG 8,17 29743 368673055 >>> /archive/index/_5fr.cfs (deleted) >>> java 19606 root 1897r REG 8,17 273561 368673054 >>> /archive/index/_5fq.cfs (deleted) >>> java 19606 root 1898r REG 8,17 9428 368673053 >>> /archive/index/_5fp.cfs (deleted) >>> java 19606 root 1899r REG 8,17 363690 368673052 >>> /archive/index/_5fo.cfs (deleted) >>> java 19606 root 1900r REG 8,17 47732 368673051 >>> /archive/index/_5fn.cfs (deleted) >>> java 19606 root 1901r REG 8,17 417453 368673050 >>> /archive/index/_5fm.cfs (deleted) >>> java 19606 root 1902r REG 8,17 1518126 368673049 >>> /archive/index/_5fl.cfs (deleted) >>> java 19606 root 1910r REG 8,17 403610 368673058 >>> /archive/index/_5fu.cfs (deleted) >>> java 19606 root 1911r REG 8,17 1236023 368673057 >>> /archive/index/_5ft.cfs (deleted) >>> java 19606 root 1912r REG 8,17 1389856 368673056 >>> /archive/index/_5fs.cfs (deleted) >>> java 19606 root 1913r REG 8,17 29743 368673055 >>> /archive/index/_5fr.cfs (deleted) >>> java 19606 root 1914r REG 8,17 273561 368673054 >>> /archive/index/_5fq.cfs (deleted) >>> java 19606 root 1915r REG 8,17 9428 368673053 >>> /archive/index/_5fp.cfs (deleted) >>> java 19606 root 1916r REG 8,17 363690 368673052 >>> /archive/index/_5fo.cfs (deleted) >>> java 19606 root 1917r REG 8,17 47732 368673051 >>> /archive/index/_5fn.cfs (deleted) >>> java 19606 root 1918r REG 8,17 417453 368673050 >>> /archive/index/_5fm.cfs (deleted) >>> java 19606 root 1919r REG 8,17 1518126 368673049 >>> /archive/index/_5fl.cfs (deleted) >>> >>> >>> -- >>> View this message in context: >>> http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25947462.html >>> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >>> For additional commands, e-mail: java-user-h...@lucene.apache.org >>> >>> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> >> >> > > -- > View this message in context: > http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25949130.html > Sent from the Lucene - Java Users mailing list archive at Nabble.com. > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
