Hi, First of all I wanted to thank the Neo4j team for the great support with this mail list.
I'm answering here to Johan Svensson reply to the other thread with the same name, (*since this is the original I started before it got replicated somehow*). >Pablo, you had a problem with batch inserter throwing an exception >upon shutdown that I suspected was due to not enough available disk >space. Then there was the the "to many open files". Are you still >experiencing problems with that? I already solved those exceptions. The first one was just that I run out of space as you said. As simply as that. I just didn't realize it since I had a volume of 300G built from a snapshot of 150G in which I forgot to run e2fsck -f and resize2fs so the partition table was not updated and it actually was like it had only 150G. Regarding the latter, as one of your team pointed out, it had something to do with the OS configuration regarding the allowed max number of files open at the same time. I managed to change the ulimit in my AWS instance and everything seems to work so far now. However one of your colleagues said that even if that could be a possible workaround, there should be something weird going on so that the BatchInserter needed to open so many files at the same time. I have to say that it's a really intensive insertion with millions of index lookups, millions of nodes and relationships insertions plus indexing nodes and relationships in the middle of all that. What do you think? Anyways I hope these two changes will make it and I won't run into more exceptions anymore. I'll let you all know how it goes once I finish this Batch Insertion step. Cheers, Pablo On Wed, Feb 16, 2011 at 10:05 AM, Pablo Pareja <[email protected]> wrote: > Hi, > > I managed to change the ulimit in my machine and didn't get the "Too many > open files" error > again. However, even though no exceptions were thrown, this morning when I > tested some > queries against the DB results were quite weird and completely unexpected. > Most nodes that should have been inserted were not, and inferring the total > number of nodes/relationships > on the DB using the call: > > * > GraphService().getConfig().getGraphDbModule().getNodeManager().getNumberOfIdsInUse(Node.class/Relationship.class) > * > * > * > I got these numbers: > nodes number = 2224916 (there should be at least 30/40.000.000) > relationships number = 286701382 > > Then I checked the messages.log file and was really surprised when I saw > this: > > Wed Feb 16 08:45:27 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore non clean > shutdown detected > Wed Feb 16 08:45:28 UTC 2011: > /mnt/bio4j_volume/bio4jdb/neostore.relationshipstore.db non clean shutdown > detected > Wed Feb 16 08:45:28 UTC 2011: > /mnt/bio4j_volume/bio4jdb/neostore.nodestore.db non clean shutdown detected > Wed Feb 16 08:45:28 UTC 2011: Opened > [/mnt/bio4j_volume/bio4jdb/nioneo_logical.log.1] clean empty log, version=0 > Wed Feb 16 08:45:28 UTC 2011: > /mnt/bio4j_volume/bio4jdb/neostore.relationshipstore.db rebuild id > generator, highId=286701382 defragged count=0 > Wed Feb 16 08:45:28 UTC 2011: > /mnt/bio4j_volume/bio4jdb/neostore.nodestore.db rebuild id generator, > highId=2224916 defragged count=0 > Wed Feb 16 08:45:28 UTC 2011: /mnt/bio4j_volume/bio4jdb/neostore rebuild id > generator, highId=4 defragged count=0 > Wed Feb 16 08:45:28 UTC 2011: Opened > [/mnt/bio4j_volume/bio4jdb/lucene/lucene.log.1] clean empty log, version=0 > Wed Feb 16 08:45:28 UTC 2011: Opened > [/mnt/bio4j_volume/bio4jdb/lucene-fulltext/lucene.log.1] clean empty log, > version=0 > Wed Feb 16 08:45:28 UTC 2011: Extension > org.neo4j.kernel.KernelExtension[kernel jmx] initialized ok > Wed Feb 16 08:45:28 UTC 2011: TM new log: tm_tx_log.1 > > How come there was a non clean shutdown() operation without any exceptions > thrown? > I'd really appreciate any help, > Regarding the boiled down version of my code I guess I could prepare it but > it's quite a big project and > it'd probably be difficult to reproduce the exact situation for the > insertion due to its dimensions. > Cheers, > > Pablo > > > On Tue, Feb 15, 2011 at 9:20 PM, Mattias Persson < > [email protected]> wrote: > >> If that many files are kept open at one point in time there's probably a >> bug >> in the code. I don't think that increasing number of files your OS allows >> is >> a correct solution, but may work as a temporary solution. I'd love to get >> a >> boiled down (or the code as is) version of you code so that I can >> replicate >> that behaviour and fix such bugs. >> >> 2011/2/15 Anders Nawroth <[email protected]> >> >> > Hi! >> > >> > 2011-02-15 13:19, Pablo Pareja: >> > > (Too many open files) >> > >> > This could be that you have old processes around that still keep lots of >> > files open. Or that you simply have to few file handles available to >> > begin with. Just google for "Too many open files" + your operating >> > system, and you'll find information on how to configure this. >> > >> > >> > /anders >> > >> > >> > > >> > >> org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.index(LuceneIndexBatchInserterImpl.java:186) >> > > >> > >> com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:467) >> > > >> > >> com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) >> > > >> > >> com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) >> > > >> > >> com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) >> > > >> > > java.lang.RuntimeException: java.io.FileNotFoundException: >> > > /mnt/bio4j_volume/bio4jdb/lucene/instit >> > ute_name_index/_c7.frq >> > > (Too many open files) >> > > at >> > > >> > >> org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:273) >> > > at >> > > >> > >> org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.shutdown(LuceneIndexBatchInserterImpl.java:214) >> > > at >> > > >> > >> com.era7.bioinfo.bio4j.programs.ImportUniprot.main(ImportUniprot.java:552) >> > > at >> > > >> > >> com.era7.bioinfo.bio4j.programs.ImportUniprot.execute(ImportUniprot.java:192) >> > > at >> > > >> > >> com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) >> > > at >> > > >> > >> com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:16) >> > > Caused by: java.io.FileNotFoundException: >> > > /mnt/bio4j_volume/bio4jdb/lucene/institute_name_index/_c7.frq (Too >> many >> > open >> > > files) >> > > at java.io.RandomAccessFile.open(Native Method) >> > > at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233) >> > > at >> > > >> > >> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexOutput.<init>(SimpleFSDirectory.java:180) >> > > at >> > > >> > >> org.apache.lucene.store.NIOFSDirectory.createOutput(NIOFSDirectory.java:74) >> > > at >> > > >> > >> org.apache.lucene.index.FormatPostingsDocsWriter.<init>(FormatPostingsDocsWriter.java:47) >> > > at >> > > >> > >> org.apache.lucene.index.FormatPostingsTermsWriter.<init>(FormatPostingsTermsWriter.java:33) >> > > at >> > > >> > >> org.apache.lucene.index.FormatPostingsFieldsWriter.<init>(FormatPostingsFieldsWriter.java:59) >> > > at >> > > >> > >> org.apache.lucene.index.FreqProxTermsWriter.flush(FreqProxTermsWriter.java:94) >> > > at >> org.apache.lucene.index.TermsHash.flush(TermsHash.java:144) >> > > at >> > org.apache.lucene.index.DocInverter.flush(DocInverter.java:72) >> > > at >> > > >> > >> org.apache.lucene.index.DocFieldProcessor.flush(DocFieldProcessor.java:64) >> > > at >> > > >> org.apache.lucene.index.DocumentsWriter.flush(DocumentsWriter.java:583) >> > > at >> > > >> > >> org.apache.lucene.index.IndexWriter.doFlushInternal(IndexWriter.java:3598) >> > > at >> > > org.apache.lucene.index.IndexWriter.doFlush(IndexWriter.java:3509) >> > > at >> > org.apache.lucene.index.IndexWriter.flush(IndexWriter.java:3500) >> > > at >> > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2271) >> > > at >> > > org.apache.lucene.index.IndexWriter.optimize(IndexWriter.java:2249) >> > > at >> > > >> > >> org.neo4j.index.lucene.LuceneIndexBatchInserterImpl.optimize(LuceneIndexBatchInserterImpl.java:267) >> > > ... 5 more >> > > >> > > I posted another question to the mail list regarding optimize() >> > indexservice >> > > which I guess could have something >> > > to do with this exception, >> > > Thanks in advance >> > > >> > > Pablo >> > > >> > > >> > > On Tue, Feb 15, 2011 at 11:57 AM, Pablo Pareja<[email protected]> >> wrote: >> > > >> > >> Ok, I remember there was a java.io.IOException afterwards, >> unfortunately >> > I >> > >> didn't copy >> > >> that part thinking it was the consequence of the previous one. >> > >> Anyways I think it said something about an index file that existed >> but >> > >> couldn't be accessed for an unknown reason, >> > >> is that possible? >> > >> I'll try to reproduce it again, >> > >> >> > >> Pablo >> > >> >> > >> >> > >> On Tue, Feb 15, 2011 at 10:45 AM, Tobias Ivarsson< >> > >> [email protected]> wrote: >> > >> >> > >>> There should have been a cause at the end of that trace, looking >> > something >> > >>> like: >> > >>> >> > >>> caused by: java.io.IOException ... >> > >>> >> > >>> That is what I'm interested in. >> > >>> >> > >>> -tobias >> > >>> >> > >>> On Tue, Feb 15, 2011 at 10:12 AM, Pablo Pareja<[email protected]> >> > wrote: >> > >>> >> > >>>> I already put the stack trace in the first message: >> > >>>> >> > >>>> Unable to write record[244986398] @[32583190934] >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >> > >>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) >> > >>>> >> > >>>> the missing lines are just from my own classes ending in a >> > >>>> BatchInserter.shutdown() method. >> > >>>> >> > >>>> Cheers, >> > >>>> >> > >>>> Pablo >> > >>>> >> > >>>> On Tue, Feb 15, 2011 at 9:06 AM, Tobias Ivarsson< >> > >>>> [email protected]> wrote: >> > >>>> >> > >>>>> I had a second look at this, and realized that those memory >> mapping >> > >>>>> settings >> > >>>>> aren't used from the batch inserter. Everything is allocated in >> the >> > >>> Java >> > >>>>> Heap. 63G out of 64G RAM is still a bit much, but not as big of a >> > >>> problem >> > >>>>> as >> > >>>>> I first thought. >> > >>>>> >> > >>>>> This also means that there could be another problem behind the >> > >>> exception >> > >>>>> you >> > >>>>> receive. There should have been a causing exception for the one >> you >> > >>>> listed. >> > >>>>> Could you please provide a full stack trace? >> > >>>>> >> > >>>>> Cheers, >> > >>>>> Tobias >> > >>>>> >> > >>>>> On Thu, Feb 10, 2011 at 10:42 PM, Pablo Pareja<[email protected]> >> > >>> wrote: >> > >>>>> >> > >>>>>> Hi all, >> > >>>>>> I keep getting this exception and don't know what the reason >> could >> > >>> be: >> > >>>>>> >> > >>>>>> Unable to write record[244986398] @[32583190934] >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:112) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:124) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:251) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:224) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:614) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:101) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:115) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:611) >> > >>>>>> >> > >>>>>> >> > >>>>> >> > >>>> >> > >>> >> > >> org.neo4j.kernel.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:351) >> > >>>>>> >> > >>>>>> I'm using a 65G RAM machine and the java process is launched with >> > >>>> -Xmx63G >> > >>>>>> the conf values I pass to the batch inserter are the following: >> > >>>>>> >> > >>>>>> neostore.nodestore.db.mapped_memory=10G >> > >>>>>> neostore.relationshipstore.db.mapped_memory=10G >> > >>>>>> neostore.propertystore.db.mapped_memory=10G >> > >>>>>> neostore.propertystore.db.strings.mapped_memory=10G >> > >>>>>> neostore.propertystore.db.arrays.mapped_memory=0M >> > >>>>>> >> > >>>>>> Besides, there were still 50G free space in the disk when this >> > >>>> exception >> > >>>>>> was >> > >>>>>> thrown >> > >>>>>> >> > >>>>>> I'd really appreciate any help, >> > >>>>>> Cheers >> > >>>>>> >> > >>>>>> -- >> > >>>>>> Pablo Pareja Tobes >> > >>>>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes >> > >>>>>> Twitter http://www.twitter.com/pablopareja >> > >>>>>> >> > >>>>>> http://www.ohnosequences.com >> > >>>>>> _______________________________________________ >> > >>>>>> Neo4j mailing list >> > >>>>>> [email protected] >> > >>>>>> https://lists.neo4j.org/mailman/listinfo/user >> > >>>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> -- >> > >>>>> Tobias Ivarsson<[email protected]> >> > >>>>> Hacker, Neo Technology >> > >>>>> www.neotechnology.com >> > >>>>> Cellphone: +46 706 534857 >> > >>>>> _______________________________________________ >> > >>>>> Neo4j mailing list >> > >>>>> [email protected] >> > >>>>> https://lists.neo4j.org/mailman/listinfo/user >> > >>>>> >> > >>>> >> > >>>> >> > >>>> >> > >>>> -- >> > >>>> Pablo Pareja Tobes >> > >>>> LinkedIn http://www.linkedin.com/in/pabloparejatobes >> > >>>> Twitter http://www.twitter.com/pablopareja >> > >>>> >> > >>>> http://www.ohnosequences.com >> > >>>> _______________________________________________ >> > >>>> Neo4j mailing list >> > >>>> [email protected] >> > >>>> https://lists.neo4j.org/mailman/listinfo/user >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> Tobias Ivarsson<[email protected]> >> > >>> Hacker, Neo Technology >> > >>> www.neotechnology.com >> > >>> Cellphone: +46 706 534857 >> > >>> _______________________________________________ >> > >>> Neo4j mailing list >> > >>> [email protected] >> > >>> https://lists.neo4j.org/mailman/listinfo/user >> > >>> >> > >> >> > >> >> > >> >> > >> -- >> > >> Pablo Pareja Tobes >> > >> LinkedIn http://www.linkedin.com/in/pabloparejatobes >> > >> Twitter http://www.twitter.com/pablopareja >> > >> >> > >> http://www.ohnosequences.com >> > >> >> > >> >> > > >> > > >> > _______________________________________________ >> > Neo4j mailing list >> > [email protected] >> > https://lists.neo4j.org/mailman/listinfo/user >> > >> >> >> >> -- >> Mattias Persson, [[email protected]] >> Hacker, Neo Technology >> www.neotechnology.com >> _______________________________________________ >> Neo4j mailing list >> [email protected] >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Pablo Pareja Tobes > LinkedIn http://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://www.ohnosequences.com > > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://about.me/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
