Following up on this, after resolving the memory problems in my code, h2 handles my 1G transaction.
Tracking down the memory problems was enormously painful. jmap/jhat provided better information than visualvm, but they were unreliable: jhat would randomly refuse to read the heap dumps generated by jmap, and it would consistently freeze up when tracing particular objects. I suspect it was incorrectly handling circular dependencies in these cases. Are there any better tools for jvm memory debugging? On Tuesday, September 10, 2013 1:21:51 PM UTC-7, Brian Craft wrote: > > Well, that was way more informative than visualvm. The top three items, > amounting to 2G: > > num #instances #bytes class name > ---------------------------------------------- > 1: 70466 1104237992 [C > 2: 45338856 1088132544 java.lang.Float > 3: 1544435 403100896 [Ljava.lang.Object; > > I don't believe any of this is h2. There are no floats in the commit, and > the char arrays in the commit are less than 6M. So my current guess is that > I'm leaking something, though I don't understand how splitting it into > multiple transactions is allowing it to work. > > On Tuesday, September 10, 2013 12:51:00 PM UTC-7, Brian Craft wrote: >> >> I do it with a bunch of inserts. The tables already exist. It happens >> even if I use UNDO_LOG=0, which I think disables the undo log. >> >> I'll try jmap. >> >> On Tuesday, September 10, 2013 11:50:43 AM UTC-7, Thomas Mueller wrote: >>> >>> Hi, >>> >>> It depends on how you do the import. Could you get a heap histogram for >>> a large import (jmap -histo:live <pid>), and could you tell us how you do >>> the import exactly? >>> >>> If you use "create table ... as select" then it can't run out of memory >>> (as far as I know), because no undo log is needed. >>> >>> If you execute many insert or update statements, then the undo log is >>> used, and at least a few bytes are needed per record (the actual data is >>> stored to disk at some point, but the position of the data is not). So in >>> theory it could run out of memory, even thought the transaction would need >>> to be huge. >>> >>> By the way, with the new MVStore storage (not production ready yet), >>> there should no longer be a limit on the size of a transaction, as no undo >>> log is needed. >>> >>> Regards, >>> Thomas >>> >>> >>> >>> >>> >>> >>> On Tue, Sep 10, 2013 at 10:51 AM, Ryan How <[email protected]> wrote: >>> >>>> Howcome it is getting OOM on import? It shouldn't do that should it? >>>> I've imported several GB of data before, I did get OOM, but increasing >>>> heap >>>> size to around 1GB worked for me. I didn't need to go to crazy sizes... >>>> >>>> >>>> On 10/09/2013 4:32 PM, Noel Grandin wrote: >>>> >>>>> >>>>> On 2013-09-06 20:15, Brian Craft wrote: >>>>> >>>>>> I need to load about 1G of data into an existing db, while >>>>>> maintaining data coherence. Wrapping the inserts in one transaction >>>>>> results >>>>>> in out-of-memory problems in the jvm. I increased the max heap size to >>>>>> 8g >>>>>> w/o improvement. I can split it into a bunch of smaller commits, which >>>>>> works fine, but then on error I need a bunch of application code to >>>>>> delete >>>>>> the transactions which succeeded. The deletes will need their own >>>>>> transactions, which could also fail. >>>>>> >>>>>> Is there any better way to do this? >>>>>> >>>>>> >>>>> Not really. >>>>> One strategy would be to copy the DB, since with H2 it's just a single >>>>> file, and then run your import process. >>>>> If it fails, just replace the modified DB with the backup. >>>>> >>>>> >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "H2 Database" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to h2-database...@**googlegroups.com. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at >>>> http://groups.google.com/**group/h2-database<http://groups.google.com/group/h2-database> >>>> . >>>> For more options, visit >>>> https://groups.google.com/**groups/opt_out<https://groups.google.com/groups/opt_out> >>>> . >>>> >>> >>> -- You received this message because you are subscribed to the Google Groups "H2 Database" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/h2-database. For more options, visit https://groups.google.com/groups/opt_out.
