There's this hardcoded setting for batch inserter indexes that
increases the ram buffer size to increase overall performance during
batch insertion. The formula is:
double atLeast = writer.getRAMBufferSizeMB();
double heapHint =
(double)(Runtime.getRuntime().maxMemory()/(1024*1024*16));
double ramBufferSize = Math.max( atLeast, heapHint );
writer.getRAMBufferSizeMB( ramBufferSize );
It's just something which I ballparked and obviously has problems for
heaps of your size. It's a tough one to fix in your case since you
cannot parameterize that calculation. The only way would be to modify
source code and build a new jar, or wait for it to be fixed (maybe by
limiting the max amount to 2Gb as Lucene seem to suggest). Hopefully
it can be addressed soon.
2011/2/24 Pablo Pareja <[email protected]>:
> Hi all,
>
> I just got this exception:
>
> ramBufferSize 3584.0 is too large; should be comfortably less than 2048
> SEVERE:
> org.apache.lucene.index.IndexWriter.setRAMBufferSizeMB(IndexWriter.java:1368)
> SEVERE:
> org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.instantiateWriter(LuceneBatchInserterIndex.java:209)
> SEVERE:
> org.neo4j.index.impl.lucene.LuceneBatchInserterIndex.<init>(LuceneBatchInserterIndex.java:66)
> SEVERE:
> org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.index(LuceneBatchInserterIndexProvider.java:135)
> SEVERE:
> org.neo4j.index.impl.lucene.LuceneBatchInserterIndexProvider.nodeIndex(LuceneBatchInserterIndexProvider.java:102)
> SEVERE:
> com.era7.bioinfo.bio4j.programs.ImportGeneOntology.main(ImportGeneOntology.java:107)
> SEVERE:
> com.era7.bioinfo.bio4j.programs.ImportGeneOntology.execute(ImportGeneOntology.java:75)
> SEVERE:
> com.era7.lib.bioinfo.bioinfoutil.ExecuteFromFile.main(ExecuteFromFile.java:66)
> SEVERE:
> com.era7.bioinfo.bio4j.programs.CreateWholeBio4j.main(CreateWholeBio4j.java:28)
>
> when launching my Batch insertion program once I had updated the indexation
> to the new API
> any ideas about what should I do?
>
> I'm launching the program in a 68GB RAM machine with the following
> batchInserter.properties:
>
> neostore.nodestore.db.mapped_memory=10G
> neostore.relationshipstore.db.mapped_memory=12G
> neostore.propertystore.db.mapped_memory=12G
> neostore.propertystore.db.strings.mapped_memory=10G
> neostore.propertystore.db.arrays.mapped_memory=512M
>
> I don't know how or where to change that "ramBufferSize' value, which by the
> way didn't specified at any point.
>
> Thanks in advance,
>
> Pablo
>
>
> --
> Pablo Pareja Tobes
> LinkedIn http://www.linkedin.com/in/pabloparejatobes
> Twitter http://www.twitter.com/pablopareja
>
> http://about.me/pablopareja
> http://www.ohnosequences.com
> _______________________________________________
> Neo4j mailing list
> [email protected]
> https://lists.neo4j.org/mailman/listinfo/user
>
--
Mattias Persson, [[email protected]]
Hacker, Neo Technology
www.neotechnology.com
_______________________________________________
Neo4j mailing list
[email protected]
https://lists.neo4j.org/mailman/listinfo/user
