Hm, I have no idea. Does h2 hold a whole transaction in memory? If you've had success with 1G transactions perhaps there's something different about my schema (larger indexes?), or perhaps I have a leak of some sort that is resolved by breaking it into smaller transactions. I could try again w/o indexes. I spent a bit of time in the visualvm memory profiler, but nothing much jumped out at me. I'm a neophyte at visualvm, though, so could have missed it.
I'd be very happy to just use a transaction instead of the thing I'm doing now, because my current scheme does not allow readers to continue accessing the old data while the load is happening. I'll have to load under a different key and mop up any relations afterward, or use a join table connecting the primary key to the latest loaded version. On Tuesday, September 10, 2013 1:51:33 AM UTC-7, Kartweel wrote: > > Howcome it is getting OOM on import? It shouldn't do that should it? > I've imported several GB of data before, I did get OOM, but increasing > heap size to around 1GB worked for me. I didn't need to go to crazy > sizes... > > On 10/09/2013 4:32 PM, Noel Grandin wrote: > > > > On 2013-09-06 20:15, Brian Craft wrote: > >> I need to load about 1G of data into an existing db, while > >> maintaining data coherence. Wrapping the inserts in one transaction > >> results in out-of-memory problems in the jvm. I increased the max > >> heap size to 8g w/o improvement. I can split it into a bunch of > >> smaller commits, which works fine, but then on error I need a bunch > >> of application code to delete the transactions which succeeded. The > >> deletes will need their own transactions, which could also fail. > >> > >> Is there any better way to do this? > >> > > > > Not really. > > One strategy would be to copy the DB, since with H2 it's just a single > > file, and then run your import process. > > If it fails, just replace the modified DB with the backup. > > > > > -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/h2-database. For more options, visit https://groups.google.com/groups/opt_out.
