I've introduced a limit so that this problem won't occur no matter how big your heap is.
2011/2/24 Pablo Pareja <[email protected]> > Cool, I thought I'd have to decrease the heap size much more. > Anyways, thanks a lot for the quick response and support. > Cheers, > > Pablo > > On Thu, Feb 24, 2011 at 3:44 PM, Mattias Persson > <[email protected]>wrote: > > > Using the formula I supplied in my previous reply I'd say you can go > > with a 32G heap or just slightly lower. > > > > 2011/2/24 Pablo Pareja <[email protected]>: > > > Hi Mattias, > > > > > > I guess that my only option would then be limiting the max amount to > 2GB > > as > > > you say, > > > however this will make the insertion phase way longer than what I > > expected > > > and counted on. > > > Just to confirm, am I then supposed to launch my java process with at > > most > > > -Xmx2G ?? > > > > > > I really think this is an important flaw to work on. > > > I look forward to seeing it fixed in the near future. > > > > > > Cheers, > > > > > > Pablo > > > > > > > > > On Thu, Feb 24, 2011 at 3:19 PM, Mattias Persson > > > <[email protected]>wrote: > > > > > >> There's this hardcoded setting for batch inserter indexes that > > >> increases the ram buffer size to increase overall performance during > > >> batch insertion. The formula is: > > >> > > >> double atLeast = writer.getRAMBufferSizeMB(); > > >> double heapHint = > > >> (double)(Runtime.getRuntime().maxMemory()/(1024*1024*16)); > > >> double ramBufferSize = Math.max( atLeast, heapHint ); > > >> writer.getRAMBufferSizeMB( ramBufferSize ); > > >> > > >> It's just something which I ballparked and obviously has problems for > > >> heaps of your size. It's a tough one to fix in your case since you > > >> cannot parameterize that calculation. The only way would be to modify > > >> source code and build a new jar, or wait for it to be fixed (maybe by > > >> limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully > > >> it can be addressed soon. > > >> > > >> 2011/2/24 Pablo Pareja <[email protected]>: > > >> > Hi all, > > >> > > > >> > I just got this exception: > > >> > > > >> > ramBufferSize 3584.0 is too large; should be comfortably less than > > 2048 > > >> > SEVERE: > > >> > > > >> > > > org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368) > > >> > SEVERE: > > >> > > > >> > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209) > > >> > SEVERE: > > >> > > > >> > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.<init>(LuceneBatchInserterIndex.java:66) > > >> > SEVERE: > > >> > > > >> > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135) > > >> > SEVERE: > > >> > > > >> > > > org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102) > > >> > SEVERE: > > >> > > > >> > > > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107) > > >> > SEVERE: > > >> > > > >> > > > com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75) > > >> > SEVERE: > > >> > > > >> > > > com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66) > > >> > SEVERE: > > >> > > > >> > > > com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28) > > >> > > > >> > when launching my Batch insertion program once I had updated the > > >> indexation > > >> > to the new API > > >> > any ideas about what should I do? > > >> > > > >> > I'm launching the program in a 68GB RAM machine with the following > > >> > batchInserter.properties: > > >> > > > >> > neostore.nodestore.db.mapped_memory=10G > > >> > neostore.relationshipstore.db.mapped_memory=12G > > >> > neostore.propertystore.db.mapped_memory=12G > > >> > neostore.propertystore.db.strings.mapped_memory=10G > > >> > neostore.propertystore.db.arrays.mapped_memory=512M > > >> > > > >> > I don't know how or where to change that "ramBufferSize' value, > which > > by > > >> the > > >> > way didn't specified at any point. > > >> > > > >> > Thanks in advance, > > >> > > > >> > Pablo > > >> > > > >> > > > >> > -- > > >> > Pablo Pareja Tobes > > >> > LinkedIn http://www.linkedin.com/in/pabloparejatobes > > >> > Twitter http://www.twitter.com/pablopareja > > >> > > > >> > http://about.me/pablopareja > > >> > http://www.ohnosequences.com > > >> > _______________________________________________ > > >> > Neo4j mailing list > > >> > [email protected] > > >> > https://lists.neo4j.org/mailman/listinfo/user > > >> > > > >> > > >> > > >> > > >> -- > > >> Mattias Persson, [[email protected]] > > >> Hacker, Neo Technology > > >> www.neotechnology.com > > >> _______________________________________________ > > >> Neo4j mailing list > > >> [email protected] > > >> https://lists.neo4j.org/mailman/listinfo/user > > >> > > > > > > > > > > > > -- > > > Pablo Pareja Tobes > > > LinkedIn http://www.linkedin.com/in/pabloparejatobes > > > Twitter http://www.twitter.com/pablopareja > > > > > > http://about.me/pablopareja > > > http://www.ohnosequences.com > > > _______________________________________________ > > > Neo4j mailing list > > > [email protected] > > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > > > > > > -- > > Mattias Persson, [[email protected]] > > Hacker, Neo Technology > > www.neotechnology.com > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > Pablo Pareja Tobes > LinkedIn http://www.linkedin.com/in/pabloparejatobes > Twitter http://www.twitter.com/pablopareja > > http://about.me/pablopareja > http://www.ohnosequences.com > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
