I confess I had not investigated the batch inserter. From the description it fits my requirements exactly.
With respect to auto-commits, it seems there are two use cases. The first is every day operations that might run out of memory. In this case it might be nice for neo4j to swap out memory to temporary disk as needed. If this performs acceptably, I think that should be default behavior. The second case is the initial population of a graph, where there is no need for roll back and so there is no need to commit to a temporary location. In this case, it seems having neo4j decide when to commit would be ideal. My concern with the first use case is that swapping to temporary storage at ideal intervals may be less efficient than having the user commit to permanent storage at less-than-ideal intervals. If that is the case, then the only real justification for committing to temporary storage would be if there was a requirement to potentially roll back a transaction that was larger than memory could support. -Paul -----Original Message----- From: [email protected] [mailto:[email protected]] On Behalf Of Mattias Persson Sent: Friday, July 09, 2010 7:30 AM To: Neo4j user discussions Subject: Re: [Neo4j] OutOfMemory while populating large graph 2010/7/9 Marko Rodriguez <[email protected]> > Hi, > > > Would it actually be worth something to be able to begin a transaction > which > > auto-committs stuff every X write operation, like a batch inserter mode > > which can be used in normal EmbeddedGraphDatabase? Kind of like: > > > > graphDb.beginTx( Mode.BATCH_INSERT ) > > > > ...so that you can start such a transaction and then just insert data > > without having to care about restarting it now and then? > > Thats cool! Does that already exist? In my code (like others on the list it > seems) I have a counter++ that every 20,000 inserts (some made up number > that is not going to throw an OutOfMemory) commits and the reopens a new > transaction. Sorta sux. > No it doesn't, I just wrote stuff which I though someone could think of as useful. A cool thing with just telling it to do a batch insert mode transaction (not the actual commit interval) is that it could look at how much memory it had to play around with and commit whenever it would be the most efficient, even having the ability to change the limit on the fly if the memory suddenly ran out. > Thanks, > Marko. > > _______________________________________________ > Neo4j mailing list > [email protected] > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [[email protected]] Hacker, Neo Technology www.neotechnology.com _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user _______________________________________________ Neo4j mailing list [email protected] https://lists.neo4j.org/mailman/listinfo/user
