Using the formula I supplied in my previous reply I'd say you can go with a 32G heap or just slightly lower.
2011/2/24 Pablo Pareja <[email protected]>: > Hi Mattias, > > I guess that my only option would then be limiting the max amount to 2GB as > you say, > however this will make the insertion phase way longer than what I expected > and counted on. > Just to confirm, am I then supposed to launch my java process with at most > -Xmx2G ?? > > I really think this is an important flaw to work on. > I look forward to seeing it fixed in the near future. > > Cheers, > > Pablo > > > On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson > <[email protected]>wrote: > >> There's this hardcoded setting for batch inserter indexes that >> increases the ram buffer size to increase overall performance during >> batch insertion. The formula is: >> >> double atLeast = writer.getRAMBufferSizeMB(); >> double heapHint = >> (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16)); >> double ramBufferSize = Math.max( atLeast, heapHint ); >> writer.getRAMBufferSizeMB( ramBufferSize ); >> >> It's just something which I ballparked and obviously has problems for >> heaps of your size. It's a tough one to fix in your case since you >> cannot parameterize that calculation. The only way would be to modify >> source code and build a new jar, or wait for it to be fixed (maybe by >> limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully >> it can be addressed soon. >> >> 2011/2/24 Pablo Pareja <[email protected]>: >> > Hi all, >> > >> > I just got this exception: >> > >> > ramBufferSize 3584.0 is too large; should be comfortably less than 2048 >> > SEVERE: >> > >> org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368) >> > SEVERE: >> > >> org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209) >> > SEVERE: >> > >> org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.<init>(LuceneBatchInserterIndex.java:66) >> > SEVERE: >> > >> org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135) >> > SEVERE: >> > >> org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102) >> > SEVERE: >> > >> com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107) >> > SEVERE: >> > >> com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75) >> > SEVERE: >> > >> com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) >> > SEVERE: >> > >> com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28) >> > >> > when launching my Batch insertion program once I had updated the >> indexation >> > to the new API >> > any ideas about what should I do? >> > >> > I'm launching the program in a 68GB RAM machine with the following >> > batchInserter.properties: >> > >> > neostore.nodestore.db.mapped_memory=10G >> > neostore.relationshipstore.db.mapped_memory=12G >> > neostore.propertystore.db.mapped_memory=12G >> > neostore.propertystore.db.strings.mapped_memory=10G >> > neostore.propertystore.db.arrays.mapped_memory=512M >> > >> > I don't know how or where to change that "ramBufferSize' value, which by >> the >> > way didn't specified at any point. >> > >> > Thanks in advance, >> > >> > Pablo >> > >> > >> > -- >> > Pablo Pareja Tobes >> > LinkedIn http://www.linkedin.com/in/pabloparejatobes >> > Twitter http://www.twitter.com/pablopareja >> > >> > http://about.me/pablopareja >> > http://www.ohnosequences.com >> > _______________________________________________ >> > Neo4j mailing list >> > [email protected] >> > https://lists.neo4j.org/mailman/listinfo/user >> > >> >> >> >> -- >> Mattias Persson, [[email protected]] >> Hacker, Neo Technology >> www.neotechnology.com >> _______________________________________________ >> Neo4j mailing list >> [email protected] >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Pablo Pareja Tobes > LinkedIn http://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://about.me/pablopareja > http://www.ohnosequences.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
