Hi, I managed to change the ulimit in my machine and didn't get the "Too many open files" error again. However, even though no exceptions were thrown, this morning when I tested some queries against the DB results were quite weird and completely unexpected. Most nodes that should have been inserted were not, and inferring the total number of nodes/relationships on the DB using the call:
* GraphService().getConfig().getGraphDbModule().getNodeManager().getNumberOfIdsInUse(Node.class/Relationship.class) * * * I got these numbers: nodes number = 2224916 (there should be at least 30/40.000.000) relationships number = 286701382 Then I checked the messages.log file and was really surprised when I saw this: Wed Feb 16 08:45:27 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore non clean shutdown detected Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore.relationshipstore.db non clean shutdown detected Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore.nodestore.db non clean shutdown detected Wed Feb 16 08:45:28 UTC 2011: Opened [/mnt/bio4j_volume/bio4jdb/nioneo_logical.log.1] clean empty log, version=0 Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore.relationshipstore.db rebuild id generator, highId=286701382 defragged count=0 Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore.nodestore.db rebuild id generator, highId=2224916 defragged count=0 Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore rebuild id generator, highId=4 defragged count=0 Wed Feb 16 08:45:28 UTC 2011: Opened [/mnt/bio4j_volume/bio4jdb/lucene/lucene.log.1] clean empty log, version=0 Wed Feb 16 08:45:28 UTC 2011: Opened [/mnt/bio4j_volume/bio4jdb/lucene-fulltext/lucene.log.1] clean empty log, version=0 Wed Feb 16 08:45:28 UTC 2011: Extension org.neo4j.kernel.KernelExtension[kernel jmx] initialized ok Wed Feb 16 08:45:28 UTC 2011: TM new log: tm_tx_log.1 How come there was a non clean shutdown() operation without any exceptions thrown? I'd really appreciate any help, Regarding the boiled down version of my code I guess I could prepare it but it's quite a big project and it'd probably be difficult to reproduce the exact situation for the insertion due to its dimensions. Cheers, Pablo On Tue, Feb 15, 2011 at 9:20 PM, Mattias Persson <[email protected]>wrote: > If that many files are kept open at one point in time there's probably a > bug > in the code. I don't think that increasing number of files your OS allows > is > a correct solution, but may work as a temporary solution. I'd love to get a > boiled down (or the code as is) version of you code so that I can replicate > that behaviour and fix such bugs. > > 2011/2/15 Anders Nawroth <[email protected]> > > > Hi! > > > > 2011-02-15 13:19, Pablo Pareja: > > > (Too many open files) > > > > This could be that you have old processes around that still keep lots of > > files open. Or that you simply have to few file handles available to > > begin with. Just google for "Too many open files" + your operating > > system, and you'll find information on how to configure this. > > > > > > /anders > > > > > > > > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.java:186) > > > > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:467) > > > > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > > > > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > > > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > > > > > > java.lang.RuntimeException: java.io.FileNotFoundException: > > > /mnt/bio4j_volume/bio4jdb/lucene/instit > > ute_name_index/_c7.frq > > > (Too many open files) > > > at > > > > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:273) > > > at > > > > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.shutdown(LuceneIndexBatchInserterImpl.java:214) > > > at > > > > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:552) > > > at > > > > > > com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) > > > at > > > > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > > at > > > > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) > > > Caused by: java.io.FileNotFoundException: > > > /mnt/bio4j_volume/bio4jdb/lucene/institute_name_index/_c7.frq (Too many > > open > > > files) > > > at java.io.RandomAccessFile.open(Native Method) > > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) > > > at > > > > > > org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) > > > at > > > > > > org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) > > > at > > > > > > org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47) > > > at > > > > > > org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33) > > > at > > > > > > org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59) > > > at > > > > > > org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:94) > > > at org.apache.lucene.index.TermsHash.flush(TermsHash.java:144) > > > at > > org.apache.lucene.index.DocInverter.flush(DocInverter.java:72) > > > at > > > > > > org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64) > > > at > > > org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583) > > > at > > > > > > org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3598) > > > at > > > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3509) > > > at > > org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3500) > > > at > > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2271) > > > at > > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2249) > > > at > > > > > > org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:267) > > > ... 5 more > > > > > > I posted another question to the mail list regarding optimize() > > indexservice > > > which I guess could have something > > > to do with this exception, > > > Thanks in advance > > > > > > Pablo > > > > > > > > > On Tue, Feb 15, 2011 at 11:57 AM, Pablo Pareja<[email protected]> > wrote: > > > > > >> Ok, I remember there was a java.io.IOException afterwards, > unfortunately > > I > > >> didn't copy > > >> that part thinking it was the consequence of the previous one. > > >> Anyways I think it said something about an index file that existed but > > >> couldn't be accessed for an unknown reason, > > >> is that possible? > > >> I'll try to reproduce it again, > > >> > > >> Pablo > > >> > > >> > > >> On Tue, Feb 15, 2011 at 10:45 AM, Tobias Ivarsson< > > >> [email protected]> wrote: > > >> > > >>> There should have been a cause at the end of that trace, looking > > something > > >>> like: > > >>> > > >>> caused by: java.io.IOException ... > > >>> > > >>> That is what I'm interested in. > > >>> > > >>> -tobias > > >>> > > >>> On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja<[email protected]> > > wrote: > > >>> > > >>>> I already put the stack trace in the first message: > > >>>> > > >>>> Unable to write record[244986398] @[32583190934] > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > >>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > >>>> > > >>>> the missing lines are just from my own classes ending in a > > >>>> BatchInserter.shutdown() method. > > >>>> > > >>>> Cheers, > > >>>> > > >>>> Pablo > > >>>> > > >>>> On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson< > > >>>> [email protected]> wrote: > > >>>> > > >>>>> I had a second look at this, and realized that those memory mapping > > >>>>> settings > > >>>>> aren't used from the batch inserter. Everything is allocated in the > > >>> Java > > >>>>> Heap. 63G out of 64G RAM is still a bit much, but not as big of a > > >>> problem > > >>>>> as > > >>>>> I first thought. > > >>>>> > > >>>>> This also means that there could be another problem behind the > > >>> exception > > >>>>> you > > >>>>> receive. There should have been a causing exception for the one you > > >>>> listed. > > >>>>> Could you please provide a full stack trace? > > >>>>> > > >>>>> Cheers, > > >>>>> Tobias > > >>>>> > > >>>>> On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja<[email protected]> > > >>> wrote: > > >>>>> > > >>>>>> Hi all, > > >>>>>> I keep getting this exception and don't know what the reason could > > >>> be: > > >>>>>> > > >>>>>> Unable to write record[244986398] @[32583190934] > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > > org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) > > >>>>>> > > >>>>>> I'm using a 65G RAM machine and the java process is launched with > > >>>> -Xmx63G > > >>>>>> the conf values I pass to the batch inserter are the following: > > >>>>>> > > >>>>>> neostore.nodestore.db.mapped_memory=10G > > >>>>>> neostore.relationshipstore.db.mapped_memory=10G > > >>>>>> neostore.propertystore.db.mapped_memory=10G > > >>>>>> neostore.propertystore.db.strings.mapped_memory=10G > > >>>>>> neostore.propertystore.db.arrays.mapped_memory=0M > > >>>>>> > > >>>>>> Besides, there were still 50G free space in the disk when this > > >>>> exception > > >>>>>> was > > >>>>>> thrown > > >>>>>> > > >>>>>> I'd really appreciate any help, > > >>>>>> Cheers > > >>>>>> > > >>>>>> -- > > >>>>>> Pablo Pareja Tobes > > >>>>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes > > >>>>>> Twitter http://www.twitter.com/pablopareja > > >>>>>> > > >>>>>> http://www.ohnosequences.com > > >>>>>> _______________________________________________ > > >>>>>> Neo4j mailing list > > >>>>>> [email protected] > > >>>>>> https://lists.neo4j.org/mailman/listinfo/user > > >>>>>> > > >>>>> > > >>>>> > > >>>>> > > >>>>> -- > > >>>>> Tobias Ivarsson<[email protected]> > > >>>>> Hacker, Neo Technology > > >>>>> www.neotechnology.com > > >>>>> Cellphone: +46 706 534857 > > >>>>> _______________________________________________ > > >>>>> Neo4j mailing list > > >>>>> [email protected] > > >>>>> https://lists.neo4j.org/mailman/listinfo/user > > >>>>> > > >>>> > > >>>> > > >>>> > > >>>> -- > > >>>> Pablo Pareja Tobes > > >>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes > > >>>> Twitter http://www.twitter.com/pablopareja > > >>>> > > >>>> http://www.ohnosequences.com > > >>>> _______________________________________________ > > >>>> Neo4j mailing list > > >>>> [email protected] > > >>>> https://lists.neo4j.org/mailman/listinfo/user > > >>>> > > >>> > > >>> > > >>> > > >>> -- > > >>> Tobias Ivarsson<[email protected]> > > >>> Hacker, Neo Technology > > >>> www.neotechnology.com > > >>> Cellphone: +46 706 534857 > > >>> _______________________________________________ > > >>> Neo4j mailing list > > >>> [email protected] > > >>> https://lists.neo4j.org/mailman/listinfo/user > > >>> > > >> > > >> > > >> > > >> -- > > >> Pablo Pareja Tobes > > >> LinkedIn http://www.linkedin.com/in/pabloparejatobes > > >> Twitter http://www.twitter.com/pablopareja > > >> > > >> http://www.ohnosequences.com > > >> > > >> > > > > > > > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Mattias Persson, [[email protected]] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
