Hi,

I solved the problem:
MySP.Record was a 3rd party object w/o Persistance. I did not want to modify it, therefore I MixedIn the Persistance class into the Record class. But not everything...


So, finally, I 'hacked' the module holding Record class (simply giving Persistance class as a its base class) and now everything seems to OK memory consuption stopped around 300M. I over the population of the 100,000th object.

Regards,
Tamas

Tamas Hegedus wrote:
Hi,

I am using ZODB3-3.5.1. I would like to populate it with large number of objects (n>100,000). I try to use subtransaction/savepoint to save memory after every 10,000th object (my objects have nested objects)

I see the increase of the size of the Data.fs file after commits, but my memory does not seem to be freed: the red line just goes upper and upper while my disk cash (swap) also started to be used heavily... I have 1G RAM...

The size of the file I parse into objects is almost 798M, approx with 200,000 records.

Parts of the code I use listed below (I tried different combinations of commit/savepoint/etc; but I do not really understand the whole picture...).

Thanks for your help and for any suggestion,
Tamas

#==============================================================
db               = ZODB.config.databaseFromURL("etc/zodb.conf")
connection       = db.open()
droot            = connection.root()
droot['MyObjs']  = OOBTree()
myDb             = droot['MyObjs']

#--------------------------------------------------------------
MixIn( MySP.Record, Persistent)

it = MySP.Iterator( open( 'dat_file.txt'),
                     MySP.RecordParser())
i  = 0

for rec in it:
    id       = copy.deepcopy( rec.ids[0])
    myDb[id] = rec
    i += 1
    if i % 10000 == 0:
        transaction.savepoint(True)
    #transaction.commit()
    #transaction.manager.free( transaction.get())
        print i

transaction.commit()
connection.close()
#==============================================================


--
Tamas Hegedus, PhD          | phone: (1) 919-966 0329
UNC - Biochem & Biophys     | fax:   (1) 919-966 5178
5007A Thurston-Bowles Bldg  | mailto:[EMAIL PROTECTED]
Chapel Hill, NC, 27599-7248 | http://biohegedus.org
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to