Cool, I thought I'd have to decrease the heap size much more. Anyways, thanks a lot for the quick response and support. Cheers,
Pablo On Thu, Feb 24, 2011 at 3:44 PM, Mattias Persson <[email protected]>wrote: > Using the formula I supplied in my previous reply I'd say you can go > with a 32G heap or just slightly lower. > > 2011/2/24 Pablo Pareja <[email protected]>: > > Hi Mattias, > > > > I guess that my only option would then be limiting the max amount to 2GB > as > > you say, > > however this will make the insertion phase way longer than what I > expected > > and counted on. > > Just to confirm, am I then supposed to launch my java process with at > most > > -Xmx2G ?? > > > > I really think this is an important flaw to work on. > > I look forward to seeing it fixed in the near future. > > > > Cheers, > > > > Pablo > > > > > > On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson > > <[email protected]>wrote: > > > >> There's this hardcoded setting for batch inserter indexes that > >> increases the ram buffer size to increase overall performance during > >> batch insertion. The formula is: > >> > >> double atLeast = writer.getRAMBufferSizeMB(); > >> double heapHint = > >> (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16)); > >> double ramBufferSize = Math.max( atLeast, heapHint ); > >> writer.getRAMBufferSizeMB( ramBufferSize ); > >> > >> It's just something which I ballparked and obviously has problems for > >> heaps of your size. It's a tough one to fix in your case since you > >> cannot parameterize that calculation. The only way would be to modify > >> source code and build a new jar, or wait for it to be fixed (maybe by > >> limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully > >> it can be addressed soon. > >> > >> 2011/2/24 Pablo Pareja <[email protected]>: > >> > Hi all, > >> > > >> > I just got this exception: > >> > > >> > ramBufferSize 3584.0 is too large; should be comfortably less than > 2048 > >> > SEVERE: > >> > > >> > org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368) > >> > SEVERE: > >> > > >> > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209) > >> > SEVERE: > >> > > >> > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.<init>(LuceneBatchInserterIndex.java:66) > >> > SEVERE: > >> > > >> > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135) > >> > SEVERE: > >> > > >> > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102) > >> > SEVERE: > >> > > >> > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107) > >> > SEVERE: > >> > > >> > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75) > >> > SEVERE: > >> > > >> > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > >> > SEVERE: > >> > > >> > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28) > >> > > >> > when launching my Batch insertion program once I had updated the > >> indexation > >> > to the new API > >> > any ideas about what should I do? > >> > > >> > I'm launching the program in a 68GB RAM machine with the following > >> > batchInserter.properties: > >> > > >> > neostore.nodestore.db.mapped_memory=10G > >> > neostore.relationshipstore.db.mapped_memory=12G > >> > neostore.propertystore.db.mapped_memory=12G > >> > neostore.propertystore.db.strings.mapped_memory=10G > >> > neostore.propertystore.db.arrays.mapped_memory=512M > >> > > >> > I don't know how or where to change that "ramBufferSize' value, which > by > >> the > >> > way didn't specified at any point. > >> > > >> > Thanks in advance, > >> > > >> > Pablo > >> > > >> > > >> > -- > >> > Pablo Pareja Tobes > >> > LinkedIn http://www.linkedin.com/in/pabloparejatobes > >> > Twitter http://www.twitter.com/pablopareja > >> > > >> > http://about.me/pablopareja > >> > http://www.ohnosequences.com > >> > _______________________________________________ > >> > Neo4j mailing list > >> > [email protected] > >> > https://lists.neo4j.org/mailman/listinfo/user > >> > > >> > >> > >> > >> -- > >> Mattias Persson, [[email protected]] > >> Hacker, Neo Technology > >> www.neotechnology.com > >> _______________________________________________ > >> Neo4j mailing list > >> [email protected] > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > > -- > > Pablo Pareja Tobes > > LinkedIn http://www.linkedin.com/in/pabloparejatobes > > Twitter http://www.twitter.com/pablopareja > > > > http://about.me/pablopareja > > http://www.ohnosequences.com > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Mattias Persson, [[email protected]] > Hacker, Neo Technology > www.neotechnology.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://about.me/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
