On a side note, Cayenne 3.0 has a leak-free DataContext context:
http://cayenne.apache.org/doc/objectcontext-memory-management.html
Andrus
On Jul 23, 2007, at 4:44 PM, Aristedes Maniatis wrote:
On 23/07/2007, at 11:29 PM, Jean-Paul Le Fèvre wrote:
Hi,
I'm trying to import a pretty big amount of data into my database.
The input is a xml formatted file. It describes more than 10 millions
of objects each having tens of attributes. The application parses
the input
file, creates the cayenne objects and commits the changes if
requested.
We've just written something quite similar. We tested it with
something like 500,000 objects across several tables.
As you can imagining I'm facing difficulties trying to avoid out
of memory
errors. Unfortunately, at this point, I'm still unable to load my big
input file.
As a starting point I hope you are using a SAX parser for your XML.
To figure out what it's happening I'm monitoring the application
behavior
with jconsole. My tactic is the following : every 10000 objects
(this number
is a parameter) I call rollbackChanges() or commitChanges().
We committed each object individually (or sometimes a logical group
of objects). That way we could have very fine control and
validation errors only caused the loss of that particular record
(or small group). I don't know that it helps to commit in batches
like this.
When I run the program in rollback mode It turns out that the
memory used
oscillates between a min and a max value as expected : after each
rollback
the garbage collector feels free to cleanup the memory.
But in commit mode the amount of memory keeps on increasing and the
application fails eventually.
Probably because the context continues to fill up with the objects
you are committing. They aren't discarded. Try creating a new
context (and discarding the old for the gc to clean up) after every
couple thousand records.
In our situation we were able to get away with 256Mb RAM on the
client (we are running this as ROP) and 512Mb RAM on the server
(most of which appears to be used by Derby).
Ari
-------------------------->
Aristedes Maniatis
phone +61 2 9660 9700
PGP fingerprint 08 57 20 4B 80 69 59 E2 A9 BF 2D 48 C2 20 0C C8
