Hi Mark, Thanks for your contribution. I'm using a m2.4xlarge EC2 AWS instance type (*68.4 GB of memory, 26 EC2 Compute Units (8 virtual cores with 3.25 EC2 Compute Units each)).* Besides I'm parsing a 17G xml file for populating the DB and normally problems only arise when I have already inserted a decent amount of data (around 30.000.000 nodes and 50.000.000 relationships), that's why I was saying it'd be kind of harsh to reproduce the same situation. Right now I'm gonna try splitting the 17G xml file in at least 10-20 files and doing that number of separated batch insertions hoping it would work that way. I'll let you all know how it goes,
Cheers, Pablo On Wed, Feb 16, 2011 at 11:12 AM, Mark Harwood <[email protected]>wrote: > Hi Pablo, > > >>Regarding the boiled down version of my code I guess I could prepare it > but it's quite a big project > > Here's a boiled-down batch load demo I did earlier based on public > Wikipedia data: http://code.google.com/p/graphdb-load-tester/ > It includes what I believe to be a faster Lucene batch indexer service > using a bloom filter and various other optimisations. > However, the underlying Neo4J database doesn't seem to be able to cope > with inserting these volumes of data on my available hardware and I > don't have (and would hope not to need) 10s of gigabytes of RAM to > throw at the problem. > > Cheers, > Mark > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
