Well, that was way more informative than visualvm. The top three items, amounting to 2G:
num #instances #bytes class name ---------------------------------------------- 1: 70466 1104237992 [C 2: 45338856 1088132544 java.lang.Float 3: 1544435 403100896 [Ljava.lang.Object; I don't believe any of this is h2. There are no floats in the commit, and the char arrays in the commit are less than 6M. So my current guess is that I'm leaking something, though I don't understand how splitting it into multiple transactions is allowing it to work. On Tuesday, September 10, 2013 12:51:00 PM UTC-7, Brian Craft wrote: > > I do it with a bunch of inserts. The tables already exist. It happens even > if I use UNDO_LOG=0, which I think disables the undo log. > > I'll try jmap. > > On Tuesday, September 10, 2013 11:50:43 AM UTC-7, Thomas Mueller wrote: >> >> Hi, >> >> It depends on how you do the import. Could you get a heap histogram for a >> large import (jmap -histo:live <pid>), and could you tell us how you do the >> import exactly? >> >> If you use "create table ... as select" then it can't run out of memory >> (as far as I know), because no undo log is needed. >> >> If you execute many insert or update statements, then the undo log is >> used, and at least a few bytes are needed per record (the actual data is >> stored to disk at some point, but the position of the data is not). So in >> theory it could run out of memory, even thought the transaction would need >> to be huge. >> >> By the way, with the new MVStore storage (not production ready yet), >> there should no longer be a limit on the size of a transaction, as no undo >> log is needed. >> >> Regards, >> Thomas >> >> >> >> >> >> >> On Tue, Sep 10, 2013 at 10:51 AM, Ryan How <[email protected]> wrote: >> >>> Howcome it is getting OOM on import? It shouldn't do that should it? >>> I've imported several GB of data before, I did get OOM, but increasing heap >>> size to around 1GB worked for me. I didn't need to go to crazy sizes... >>> >>> >>> On 10/09/2013 4:32 PM, Noel Grandin wrote: >>> >>>> >>>> On 2013-09-06 20:15, Brian Craft wrote: >>>> >>>>> I need to load about 1G of data into an existing db, while maintaining >>>>> data coherence. Wrapping the inserts in one transaction results in >>>>> out-of-memory problems in the jvm. I increased the max heap size to 8g >>>>> w/o >>>>> improvement. I can split it into a bunch of smaller commits, which works >>>>> fine, but then on error I need a bunch of application code to delete the >>>>> transactions which succeeded. The deletes will need their own >>>>> transactions, >>>>> which could also fail. >>>>> >>>>> Is there any better way to do this? >>>>> >>>>> >>>> Not really. >>>> One strategy would be to copy the DB, since with H2 it's just a single >>>> file, and then run your import process. >>>> If it fails, just replace the modified DB with the backup. >>>> >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "H2 Database" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to h2-database...@**googlegroups.com. >>> To post to this group, send email to [email protected]. >>> Visit this group at >>> http://groups.google.com/**group/h2-database<http://groups.google.com/group/h2-database> >>> . >>> For more options, visit >>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>> . >>> >> >> -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/h2-database. For more options, visit https://groups.google.com/groups/opt_out.
