If that many files are kept open at one point in time there's probably a bug in the code. I don't think that increasing number of files your OS allows is a correct solution, but may work as a temporary solution. I'd love to get a boiled down (or the code as is) version of you code so that I can replicate that behaviour and fix such bugs.
2011/2/15 Anders Nawroth <[email protected]> > Hi! > > 2011-02-15 13:19, Pablo Pareja: > > (Too many open files) > > This could be that you have old processes around that still keep lots of > files open. Or that you simply have to few file handles available to > begin with. Just google for "Too many open files" + your operating > system, and you'll find information on how to configure this. > > > /anders > > > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.java:186) > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:467) > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > > > > java.lang.RuntimeException: java.io.FileNotFoundException: > > /mnt/bio4j_volume/bio4jdb/lucene/instit > ute_name_index/_c7.frq > > (Too many open files) > > at > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:273) > > at > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.shutdown(LuceneIndexBatchInserterImpl.java:214) > > at > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:552) > > at > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > > at > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > at > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > > Caused by: java.io.FileNotFoundException: > > /mnt/bio4j_volume/bio4jdb/lucene/institute_name_index/_c7.frq (Too many > open > > files) > > at java.io.RandomAccessFile.open(Native Method) > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > > at > > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) > > at > > > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > > at > > > org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47) > > at > > > org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33) > > at > > > org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59) > > at > > > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:94) > > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:144) > > at > org.apache.lucene.index.DocInverter.flush(DocInverter.java:72) > > at > > > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64) > > at > > org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583) > > at > > > org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3598) > > at > > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3509) > > at > org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3500) > > at > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2271) > > at > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2249) > > at > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:267) > > ... 5 more > > > > I posted another question to the mail list regarding optimize() > indexservice > > which I guess could have something > > to do with this exception, > > Thanks in advance > > > > Pablo > > > > > > On Tue, Feb 15, 2011 at 11:57 AM, Pablo Pareja<[email protected]> wrote: > > > >> Ok, I remember there was a java.io.IOException afterwards, unfortunately > I > >> didn't copy > >> that part thinking it was the consequence of the previous one. > >> Anyways I think it said something about an index file that existed but > >> couldn't be accessed for an unknown reason, > >> is that possible? > >> I'll try to reproduce it again, > >> > >> Pablo > >> > >> > >> On Tue, Feb 15, 2011 at 10:45 AM, Tobias Ivarsson< > >> [email protected]> wrote: > >> > >>> There should have been a cause at the end of that trace, looking > something > >>> like: > >>> > >>> caused by: java.io.IOException ... > >>> > >>> That is what I'm interested in. > >>> > >>> -tobias > >>> > >>> On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja<[email protected]> > wrote: > >>> > >>>> I already put the stack trace in the first message: > >>>> > >>>> Unable to write record[244986398] @[32583190934] > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > >>>> > >>>> > >>> > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > >>>> > >>>> the missing lines are just from my own classes ending in a > >>>> BatchInserter.shutdown() method. > >>>> > >>>> Cheers, > >>>> > >>>> Pablo > >>>> > >>>> On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson< > >>>> [email protected]> wrote: > >>>> > >>>>> I had a second look at this, and realized that those memory mapping > >>>>> settings > >>>>> aren't used from the batch inserter. Everything is allocated in the > >>> Java > >>>>> Heap. 63G out of 64G RAM is still a bit much, but not as big of a > >>> problem > >>>>> as > >>>>> I first thought. > >>>>> > >>>>> This also means that there could be another problem behind the > >>> exception > >>>>> you > >>>>> receive. There should have been a causing exception for the one you > >>>> listed. > >>>>> Could you please provide a full stack trace? > >>>>> > >>>>> Cheers, > >>>>> Tobias > >>>>> > >>>>> On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja<[email protected]> > >>> wrote: > >>>>> > >>>>>> Hi all, > >>>>>> I keep getting this exception and don't know what the reason could > >>> be: > >>>>>> > >>>>>> Unable to write record[244986398] @[32583190934] > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > >>>>>> > >>>>>> I'm using a 65G RAM machine and the java process is launched with > >>>> -Xmx63G > >>>>>> the conf values I pass to the batch inserter are the following: > >>>>>> > >>>>>> neostore.nodestore.db.mapped_memory=10G > >>>>>> neostore.relationshipstore.db.mapped_memory=10G > >>>>>> neostore.propertystore.db.mapped_memory=10G > >>>>>> neostore.propertystore.db.strings.mapped_memory=10G > >>>>>> neostore.propertystore.db.arrays.mapped_memory=0M > >>>>>> > >>>>>> Besides, there were still 50G free space in the disk when this > >>>> exception > >>>>>> was > >>>>>> thrown > >>>>>> > >>>>>> I'd really appreciate any help, > >>>>>> Cheers > >>>>>> > >>>>>> -- > >>>>>> Pablo Pareja Tobes > >>>>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes > >>>>>> Twitter http://www.twitter.com/pablopareja > >>>>>> > >>>>>> http://www.ohnosequences.com > >>>>>> _______________________________________________ > >>>>>> Neo4j mailing list > >>>>>> [email protected] > >>>>>> https://lists.neo4j.org/mailman/listinfo/user > >>>>>> > >>>>> > >>>>> > >>>>> > >>>>> -- > >>>>> Tobias Ivarsson<[email protected]> > >>>>> Hacker, Neo Technology > >>>>> www.neotechnology.com > >>>>> Cellphone: +46 706 534857 > >>>>> _______________________________________________ > >>>>> Neo4j mailing list > >>>>> [email protected] > >>>>> https://lists.neo4j.org/mailman/listinfo/user > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Pablo Pareja Tobes > >>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes > >>>> Twitter http://www.twitter.com/pablopareja > >>>> > >>>> http://www.ohnosequences.com > >>>> _______________________________________________ > >>>> Neo4j mailing list > >>>> [email protected] > >>>> https://lists.neo4j.org/mailman/listinfo/user > >>>> > >>> > >>> > >>> > >>> -- > >>> Tobias Ivarsson<[email protected]> > >>> Hacker, Neo Technology > >>> www.neotechnology.com > >>> Cellphone: +46 706 534857 > >>> _______________________________________________ > >>> Neo4j mailing list > >>> [email protected] > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> > >> > >> > >> -- > >> Pablo Pareja Tobes > >> LinkedIn http://www.linkedin.com/in/pabloparejatobes > >> Twitter http://www.twitter.com/pablopareja > >> > >> http://www.ohnosequences.com > >> > >> > > > > > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
