|
I haven't had time to try your test. I don't see any issues straight away. If you commit more frequently (like every 1000 inserts), do you still see the issue?
If the issue goes away at that point, then the problem is most likely related to the transaction manager's dirty page list. All dirty pages are held in RAM until commit is called, and even then there are a number of committed transactions that are held in RAM to make flushing to the database file faster - by default, this is 10 transactions worth of changes. So even though you are comitting every 10K inserts, what you really have is a high watermark of 100K worth of inserts in the transaction manager.
It may be worth determining the approximate iteration number you are at when the out of memory error throws, back that off by 20%, then set a debug point at that level and take a look at TansactionManager.txns to get a feel for the total number of BlockIo objects that are being held.
- K
----------------------- Original Message -----------------------
From: Floris Ouwendijk <floris...@gmail.com>
Cc:
Date: Sat, 28 Nov 2009 22:55:19 +0100
Subject: [Jdbm-general] Out of memory error
Hi,
I'm using JDBM to store several million records, essentially mappings from a 10 byte array key to a 20 byte array value.. It seems that the system runs out of memory after a little while. I created a small testcase to demonstrate the issue, see below (where keys and values were changed for longs). I run this with a max heap size of 32M to illustrate the situation. If the keys are consecutive, the problem doesn't appear (but that might just be for the first millions). A heapdump shows that there are quite a few PhysicalRowId objects and most of the memory (90%) is allocated to byte arrays. I've tested this with the 1.0 version, as well as the latest from the SVN trunk. Any ideas on this issue? Kind regards, Floris import java.io.File; import java.io.IOException; import jdbm.RecordManager; import jdbm.RecordManagerFactory; import jdbm.btree.BTree; import jdbm.helper.LongComparator; import jdbm.helper.LongSerializer; public class Main { private static String FOLDER = "c:\\temp\\jdbm\\"; public static void main(String[] args) throws IOException{ for (File f : new File(FOLDER).listFiles()) f.delete(); RecordManager mgr = RecordManagerFactory.createRecordManager(FOLDER); BTree tree = BTree.createInstance( mgr, new LongComparator(), new LongSerializer(), new LongSerializer() ); mgr.setNamedObject( "test", tree.getRecid() ); for (long i =0; i < 2500000; i++) { tree.insert((long)(Math.random()*1000000000), i, false); if (i % 10000 == 0) mgr.commit(); } } } ------------------------------------------------------------------------------ Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july _______________________________________________ Jdbm-general mailing list Jdbm-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jdbm-general |
------------------------------------------------------------------------------ Join us December 9, 2009 for the Red Hat Virtual Experience, a free event focused on virtualization and cloud computing. Attend in-depth sessions from your desk. Your couch. Anywhere. http://p.sf.net/sfu/redhat-sfdev2dev
_______________________________________________ Jdbm-general mailing list Jdbm-general@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/jdbm-general
