Hi,

I am using ZODB3-3.5.1. I would like to populate it with large number of objects (n>100,000). I try to use subtransaction/savepoint to save memory after every 10,000th object (my objects have nested objects)


I see the increase of the size of the Data.fs file after commits, but my memory does not seem to be freed: the red line just goes upper and upper while my disk cash (swap) also started to be used heavily... I have 1G RAM...

The size of the file I parse into objects is almost 798M, approx with 200,000 records.

Parts of the code I use listed below (I tried different combinations of commit/savepoint/etc; but I do not really understand the whole picture...).

Thanks for your help and for any suggestion,
Tamas

#==============================================================
db               = ZODB.config.databaseFromURL("etc/zodb.conf")
connection       = db.open()
droot            = connection.root()
droot['MyObjs']  = OOBTree()
myDb             = droot['MyObjs']

#--------------------------------------------------------------
MixIn( MySP.Record, Persistent)

it = MySP.Iterator( open( 'dat_file.txt'),
                     MySP.RecordParser())
i  = 0

for rec in it:
    id       = copy.deepcopy( rec.ids[0])
    myDb[id] = rec
    i += 1
    if i % 10000 == 0:
        transaction.savepoint(True)
        #transaction.commit()
        #transaction.manager.free( transaction.get())
        print i

transaction.commit()
connection.close()
#==============================================================

--
Tamas Hegedus, PhD          | phone: (1) 919-966 0329
UNC - Biochem & Biophys     | fax:   (1) 919-966 5178
5007A Thurston-Bowles Bldg  | mailto:[EMAIL PROTECTED]
Chapel Hill, NC, 27599-7248 | http://biohegedus.org
_______________________________________________
For more information about ZODB, see the ZODB Wiki:
http://www.zope.org/Wikis/ZODB/

ZODB-Dev mailing list  -  ZODB-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zodb-dev

Reply via email to