Hi David, there's another thread I started with the same name: "Help with exception using BatchInserter". (*that's the one we are* *referring to)* I guess Mark started this one by mistake instead of replying to the other. Cheers,
Pablo On Wed, Feb 16, 2011 at 6:55 PM, David Montag < [email protected]> wrote: > Hi guys, > > I don't have the history of this thread. What issues are you running into? > > David > > On Wed, Feb 16, 2011 at 2:37 AM, Pablo Pareja <[email protected]> wrote: > > > Hi Mark, > > > > Thanks for your contribution. > > I'm using a m2.4xlarge EC2 AWS instance type (*68.4 GB of memory, 26 > > EC2 Compute > > Units (8 virtual cores with 3.25 EC2 Compute Units each)).* > > Besides I'm parsing a 17G xml file for populating the DB and normally > > problems only arise when I have > > already inserted a decent amount of data (around 30.000.000 nodes and > > 50.000.000 relationships), that's why > > I was saying it'd be kind of harsh to reproduce the same situation. > > Right now I'm gonna try splitting the 17G xml file in at least 10-20 > files > > and doing that number of > > separated batch insertions hoping it would work that way. > > I'll let you all know how it goes, > > > > Cheers, > > > > Pablo > > > > On Wed, Feb 16, 2011 at 11:12 AM, Mark Harwood <[email protected] > > >wrote: > > > > > Hi Pablo, > > > > > > >>Regarding the boiled down version of my code I guess I could prepare > it > > > but it's quite a big project > > > > > > Here's a boiled-down batch load demo I did earlier based on public > > > Wikipedia data: http://code.google.com/p/graphdb-load-tester/ > > > It includes what I believe to be a faster Lucene batch indexer service > > > using a bloom filter and various other optimisations. > > > However, the underlying Neo4J database doesn't seem to be able to cope > > > with inserting these volumes of data on my available hardware and I > > > don't have (and would hope not to need) 10s of gigabytes of RAM to > > > throw at the problem. > > > > > > Cheers, > > > Mark > > > _______________________________________________ > > > Neo4j mailing list > > > [email protected] > > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > > > > > > -- > > Pablo Pareja Tobes > > LinkedIn http://www.linkedin.com/in/pabloparejatobes > > Twitter http://www.twitter.com/pablopareja > > > > http://www.ohnosequences.com > > _______________________________________________ > > Neo4j mailing list > > [email protected] > > https://lists.neo4j.org/mailman/listinfo/user > > > > > > -- > David Montag > Neo Technology, www.neotechnology.com > Cell: 650.556.4411 > [email protected] > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Pablo Pareja Tobes LinkedIn http://www.linkedin.com/in/pabloparejatobes Twitter http://www.twitter.com/pablopareja http://www.ohnosequences.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
