Hi Mattias, I guess that my only option would then be limiting the max amount to 2GB as you say, however this will make the insertion phase way longer than what I expected and counted on. Just to confirm, am I then supposed to launch my java process with at most -Xmx2G ??
I really think this is an important flaw to work on. I look forward to seeing it fixed in the near future. Cheers, Pablo On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson <[email protected]>wrote: > There's this hardcoded setting for batch inserter indexes that > increases the ram buffer size to increase overall performance during > batch insertion. The formula is: > > double atLeast = writer.getRAMBufferSizeMB(); > double heapHint = > (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16)); > double ramBufferSize = Math.max( atLeast, heapHint ); > writer.getRAMBufferSizeMB( ramBufferSize ); > > It's just something which I ballparked and obviously has problems for > heaps of your size. It's a tough one to fix in your case since you > cannot parameterize that calculation. The only way would be to modify > source code and build a new jar, or wait for it to be fixed (maybe by > limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully > it can be addressed soon. > > 2011/2/24 Pablo Pareja <[email protected]>: > > Hi all, > > > > I just got this exception: > > > > ramBufferSize 3584.0 is too large; should be comfortably less than 2048 > > SEVERE: > > > org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368) > > SEVERE: > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209) > > SEVERE: > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.<init>(LuceneBatchInserterIndex.java:66) > > SEVERE: > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135) > > SEVERE: > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102) > > SEVERE: > > > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107) > > SEVERE: > > > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75) > > SEVERE: > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > SEVERE: > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28) > > > > when launching my Batch insertion program once I had updated the > indexation > > to the new API > > any ideas about what should I do? > > > > I'm launching the program in a 68GB RAM machine with the following > > batchInserter.properties: > > > > neostore.nodestore.db.mapped_memory=10G > > neostore.relationshipstore.db.mapped_memory=12G > > neostore.propertystore.db.mapped_memory=12G > > neostore.propertystore.db.strings.mapped_memory=10G > > neostore.propertystore.db.arrays.mapped_memory=512M > > > > I don't know how or where to change that "ramBufferSize' value, which by > the > > way didn't specified at any point. > > > > Thanks in advance, > > > > Pablo > > > > > > -- > > Pablo Pareja Tobes > > LinkedIn http://www.linkedin.com/in/pabloparejatobes > > Twitter http://www.twitter.com/pablopareja > > > > http://about.me/pablopareja > > http://www.ohnosequences.com > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Mattias Persson, [[email protected]] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://about.me/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
