Are you using 2.9.0 or 2.4?
Aaron McCurry wrote: > > I also have seen this problem recently. I had to make a patch to our > production system to at least relief the system of the deleted files > handles. I did it by capturing all of the Descriptors that the > FileInputIndex object creates and monitoring if the file it references > still exists or not. If it doesn't I then try and close the > Descriptors. I know that this is a hack, but it was needed ASAP, and > I couldn't find in my code anywhere where I wasn't closing the opened > indexreaders or indexwriters. > > Aaron > > > On Oct 18, 2009, at 11:14 AM, GlenAbbeyDrive wrote: > >> >> I switched from Lucene 2.4.0 to the latest 2.9.0 version and got too >> many >> files open within a few hours from my indexing process. Our >> indexing Java >> process adds about 2000 documents/minute. >> The IndexWriter (iw) has the following settings: >> >> iw.setMaxFieldLength(1024*1024*1024); // 1G as maximum >> terms. >> iw.setMergeFactor(10); >> iw.setMaxMergeDocs(100000); >> iw.setMaxBufferedDocs(1000); >> iw.setRAMBufferSizeMB(128); >> iw.setUseCompoundFile(true); >> >> >> >> The 'lsof' result shows rapidly growing number of open files in >> 'deleted' >> state. Counting the number of these: >> >> [server-1]#date >> Sun Oct 18 11:02:37 EDT 2009 >> [server-1]# lsof -p 19606 |grep deleted |wc -l >> 3187 >> [server-1]# date >> Sun Oct 18 11:07:29 EDT 2009 >> [server-1]# lsof -p 19606 |grep deleted |wc -l >> 3912 >> >> Here is a sample of those open files left behind in 'deleted' state: >> >> java 19606 root 1861r REG 8,17 273561 368673054 >> /archive/index/_5fq.cfs (deleted) >> java 19606 root 1862r REG 8,17 9428 368673053 >> /archive/index/_5fp.cfs (deleted) >> java 19606 root 1863r REG 8,17 363690 368673052 >> /archive/index/_5fo.cfs (deleted) >> java 19606 root 1864r REG 8,17 47732 368673051 >> /archive/index/_5fn.cfs (deleted) >> java 19606 root 1865r REG 8,17 417453 368673050 >> /archive/index/_5fm.cfs (deleted) >> java 19606 root 1866r REG 8,17 1518126 368673049 >> /archive/index/_5fl.cfs (deleted) >> java 19606 root 1871r REG 8,17 1236023 368673057 >> /archive/index/_5ft.cfs (deleted) >> java 19606 root 1872r REG 8,17 29743 368673055 >> /archive/index/_5fr.cfs (deleted) >> java 19606 root 1873r REG 8,17 273561 368673054 >> /archive/index/_5fq.cfs (deleted) >> java 19606 root 1874r REG 8,17 9428 368673053 >> /archive/index/_5fp.cfs (deleted) >> java 19606 root 1875r REG 8,17 363690 368673052 >> /archive/index/_5fo.cfs (deleted) >> java 19606 root 1876r REG 8,17 47732 368673051 >> /archive/index/_5fn.cfs (deleted) >> java 19606 root 1877r REG 8,17 417453 368673050 >> /archive/index/_5fm.cfs (deleted) >> java 19606 root 1878r REG 8,17 1518126 368673049 >> /archive/index/_5fl.cfs (deleted) >> java 19606 root 1883r REG 8,17 1389856 368673056 >> /archive/index/_5fs.cfs (deleted) >> java 19606 root 1884r REG 8,17 29743 368673055 >> /archive/index/_5fr.cfs (deleted) >> java 19606 root 1885r REG 8,17 273561 368673054 >> /archive/index/_5fq.cfs (deleted) >> java 19606 root 1886r REG 8,17 9428 368673053 >> /archive/index/_5fp.cfs (deleted) >> java 19606 root 1887r REG 8,17 363690 368673052 >> /archive/index/_5fo.cfs (deleted) >> java 19606 root 1888r REG 8,17 47732 368673051 >> /archive/index/_5fn.cfs (deleted) >> java 19606 root 1889r REG 8,17 417453 368673050 >> /archive/index/_5fm.cfs (deleted) >> java 19606 root 1890r REG 8,17 1518126 368673049 >> /archive/index/_5fl.cfs (deleted) >> java 19606 root 1895r REG 8,17 1389856 368673056 >> /archive/index/_5fs.cfs (deleted) >> java 19606 root 1896r REG 8,17 29743 368673055 >> /archive/index/_5fr.cfs (deleted) >> java 19606 root 1897r REG 8,17 273561 368673054 >> /archive/index/_5fq.cfs (deleted) >> java 19606 root 1898r REG 8,17 9428 368673053 >> /archive/index/_5fp.cfs (deleted) >> java 19606 root 1899r REG 8,17 363690 368673052 >> /archive/index/_5fo.cfs (deleted) >> java 19606 root 1900r REG 8,17 47732 368673051 >> /archive/index/_5fn.cfs (deleted) >> java 19606 root 1901r REG 8,17 417453 368673050 >> /archive/index/_5fm.cfs (deleted) >> java 19606 root 1902r REG 8,17 1518126 368673049 >> /archive/index/_5fl.cfs (deleted) >> java 19606 root 1910r REG 8,17 403610 368673058 >> /archive/index/_5fu.cfs (deleted) >> java 19606 root 1911r REG 8,17 1236023 368673057 >> /archive/index/_5ft.cfs (deleted) >> java 19606 root 1912r REG 8,17 1389856 368673056 >> /archive/index/_5fs.cfs (deleted) >> java 19606 root 1913r REG 8,17 29743 368673055 >> /archive/index/_5fr.cfs (deleted) >> java 19606 root 1914r REG 8,17 273561 368673054 >> /archive/index/_5fq.cfs (deleted) >> java 19606 root 1915r REG 8,17 9428 368673053 >> /archive/index/_5fp.cfs (deleted) >> java 19606 root 1916r REG 8,17 363690 368673052 >> /archive/index/_5fo.cfs (deleted) >> java 19606 root 1917r REG 8,17 47732 368673051 >> /archive/index/_5fn.cfs (deleted) >> java 19606 root 1918r REG 8,17 417453 368673050 >> /archive/index/_5fm.cfs (deleted) >> java 19606 root 1919r REG 8,17 1518126 368673049 >> /archive/index/_5fl.cfs (deleted) >> >> >> -- >> View this message in context: >> http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25947462.html >> Sent from the Lucene - Java Users mailing list archive at Nabble.com. >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org >> For additional commands, e-mail: java-user-h...@lucene.apache.org >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org > For additional commands, e-mail: java-user-h...@lucene.apache.org > > > -- View this message in context: http://www.nabble.com/Lucene-2.9.0-leaves-too-many-.cfs-files-open%2C-causing-too-many-files-open-java-error.-tp25947462p25948828.html Sent from the Lucene - Java Users mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
