Hi! 2011-02-15 13:19, Pablo Pareja: > (Too many open files)
This could be that you have old processes around that still keep lots of files open. Or that you simply have to few file handles available to begin with. Just google for "Too many open files" + your operating system, and you'll find information on how to configure this. /anders > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.java:186) > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:467) > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > > java.lang.RuntimeException: java.io.FileNotFoundException: > /mnt/bio4j_volume/bio4jdb/lucene/instit ute_name_index/_c7.frq > (Too many open files) > at > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:273) > at > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.shutdown(LuceneIndexBatchInserterImpl.java:214) > at > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:552) > at > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > at > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > at > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > Caused by: java.io.FileNotFoundException: > /mnt/bio4j_volume/bio4jdb/lucene/institute_name_index/_c7.frq (Too many open > files) > at java.io.RandomAccessFile.open(Native Method) > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > at > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) > at > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > at > org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47) > at > org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33) > at > org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59) > at > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:94) > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:144) > at org.apache.lucene.index.DocInverter.flush(DocInverter.java:72) > at > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64) > at > org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583) > at > org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3598) > at > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3509) > at org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3500) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2271) > at > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2249) > at > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:267) > ... 5 more > > I posted another question to the mail list regarding optimize() indexservice > which I guess could have something > to do with this exception, > Thanks in advance > > Pablo > > > On Tue, Feb 15, 2011 at 11:57 AM, Pablo Pareja<[email protected]> wrote: > >> Ok, I remember there was a java.io.IOException afterwards, unfortunately I >> didn't copy >> that part thinking it was the consequence of the previous one. >> Anyways I think it said something about an index file that existed but >> couldn't be accessed for an unknown reason, >> is that possible? >> I'll try to reproduce it again, >> >> Pablo >> >> >> On Tue, Feb 15, 2011 at 10:45 AM, Tobias Ivarsson< >> [email protected]> wrote: >> >>> There should have been a cause at the end of that trace, looking something >>> like: >>> >>> caused by: java.io.IOException ... >>> >>> That is what I'm interested in. >>> >>> -tobias >>> >>> On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja<[email protected]> wrote: >>> >>>> I already put the stack trace in the first message: >>>> >>>> Unable to write record[244986398] @[32583190934] >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >>>> >>> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) >>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >>>> >>>> >>> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) >>>> >>>> the missing lines are just from my own classes ending in a >>>> BatchInserter.shutdown() method. >>>> >>>> Cheers, >>>> >>>> Pablo >>>> >>>> On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson< >>>> [email protected]> wrote: >>>> >>>>> I had a second look at this, and realized that those memory mapping >>>>> settings >>>>> aren't used from the batch inserter. Everything is allocated in the >>> Java >>>>> Heap. 63G out of 64G RAM is still a bit much, but not as big of a >>> problem >>>>> as >>>>> I first thought. >>>>> >>>>> This also means that there could be another problem behind the >>> exception >>>>> you >>>>> receive. There should have been a causing exception for the one you >>>> listed. >>>>> Could you please provide a full stack trace? >>>>> >>>>> Cheers, >>>>> Tobias >>>>> >>>>> On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja<[email protected]> >>> wrote: >>>>> >>>>>> Hi all, >>>>>> I keep getting this exception and don't know what the reason could >>> be: >>>>>> >>>>>> Unable to write record[244986398] @[32583190934] >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >>>>>> >>>>>> >>>>> >>>> >>> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) >>>>>> >>>>>> I'm using a 65G RAM machine and the java process is launched with >>>> -Xmx63G >>>>>> the conf values I pass to the batch inserter are the following: >>>>>> >>>>>> neostore.nodestore.db.mapped_memory=10G >>>>>> neostore.relationshipstore.db.mapped_memory=10G >>>>>> neostore.propertystore.db.mapped_memory=10G >>>>>> neostore.propertystore.db.strings.mapped_memory=10G >>>>>> neostore.propertystore.db.arrays.mapped_memory=0M >>>>>> >>>>>> Besides, there were still 50G free space in the disk when this >>>> exception >>>>>> was >>>>>> thrown >>>>>> >>>>>> I'd really appreciate any help, >>>>>> Cheers >>>>>> >>>>>> -- >>>>>> Pablo Pareja Tobes >>>>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes >>>>>> Twitter http://www.twitter.com/pablopareja >>>>>> >>>>>> http://www.ohnosequences.com >>>>>> _______________________________________________ >>>>>> Neo4j mailing list >>>>>> [email protected] >>>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Tobias Ivarsson<[email protected]> >>>>> Hacker, Neo Technology >>>>> www.neotechnology.com >>>>> Cellphone: +46 706 534857 >>>>> _______________________________________________ >>>>> Neo4j mailing list >>>>> [email protected] >>>>> https://lists.neo4j.org/mailman/listinfo/user >>>>> >>>> >>>> >>>> >>>> -- >>>> Pablo Pareja Tobes >>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes >>>> Twitter http://www.twitter.com/pablopareja >>>> >>>> http://www.ohnosequences.com >>>> _______________________________________________ >>>> Neo4j mailing list >>>> [email protected] >>>> https://lists.neo4j.org/mailman/listinfo/user >>>> >>> >>> >>> >>> -- >>> Tobias Ivarsson<[email protected]> >>> Hacker, Neo Technology >>> www.neotechnology.com >>> Cellphone: +46 706 534857 >>> _______________________________________________ >>> Neo4j mailing list >>> [email protected] >>> https://lists.neo4j.org/mailman/listinfo/user >>> >> >> >> >> -- >> Pablo Pareja Tobes >> LinkedIn http://www.linkedin.com/in/pabloparejatobes >> Twitter http://www.twitter.com/pablopareja >> >> http://www.ohnosequences.com >> >> > > _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
