Hi,
I solved the problem:
MySP.Record was a 3rd party object w/o Persistance. I did not want to
modify it, therefore I MixedIn the Persistance class into the Record
class. But not everything...
So, finally, I 'hacked' the module holding Record class (simply giving
Persistance class as a its base class) and now everything seems to OK
memory consuption stopped around 300M. I over the population of the
100,000th object.
Regards,
Tamas
Tamas Hegedus wrote:
Hi,
I am using ZODB3-3.5.1. I would like to populate it with large number of
objects (n>100,000). I try to use subtransaction/savepoint to save
memory after every 10,000th object (my objects have nested objects)
I see the increase of the size of the Data.fs file after commits, but my
memory does not seem to be freed: the red line just goes upper and upper
while my disk cash (swap) also started to be used heavily... I have 1G
RAM...
The size of the file I parse into objects is almost 798M, approx with
200,000 records.
Parts of the code I use listed below (I tried different combinations of
commit/savepoint/etc; but I do not really understand the whole picture...).
Thanks for your help and for any suggestion,
Tamas
#==============================================================
db = ZODB.config.databaseFromURL("etc/zodb.conf")
connection = db.open()
droot = connection.root()
droot['MyObjs'] = OOBTree()
myDb = droot['MyObjs']
#--------------------------------------------------------------
MixIn( MySP.Record, Persistent)
it = MySP.Iterator( open( 'dat_file.txt'),
MySP.RecordParser())
i = 0
for rec in it:
id = copy.deepcopy( rec.ids[0])
myDb[id] = rec
i += 1
if i % 10000 == 0:
transaction.savepoint(True)
#transaction.commit()
#transaction.manager.free( transaction.get())
print i
transaction.commit()
connection.close()
#==============================================================
--
Tamas Hegedus, PhD | phone: (1) 919-966 0329
UNC - Biochem & Biophys | fax: (1) 919-966 5178
5007A Thurston-Bowles Bldg | mailto:[EMAIL PROTECTED]
Chapel Hill, NC, 27599-7248 | http://biohegedus.org
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/
ZODB-Dev mailing list - ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev