peter -
i think your approach is correct in theory, but i havent done a lot
of testing with "expunge" to insure that it cleans everything up as
its a recently added feature.
I would try to make sure it is indeed the mapper call that is growing
memory, such as by trying the below loop without using the mapper
call (maybe something else is doing it, like the plain SQL, or
something else). also try seeing if doing a session.clear() has
better results than expunge().
also my observations with CPython is that even under total memory-
cleaning conditions, the interpreter still can grow slightly.
ill take a look at this myself.
On Apr 19, 2006, at 11:03 AM, Peter L. Buschman wrote:
All:
I'm trying to process a large set of records in batches to control
memory usage, but after each batch,
the memory usage keeps going up. I suspect I am not freeing-up the
processed mapped objects correctly.
Can anyone here enlighten me as to what I'm doing wrong? Is there
another function I should be calling other
than session.expunge() once I have processed a record and am ready
to discard it? Out of the entire set,
only a small number will need to be updated so I can discard the
majority once I have determined that they
are not eligible to be changed.
The complete record set can be quite huge so I desperately need a
way to control Python's memory usage
while iterating through result sets. :(
Kindest regards,
Peter Buschman
session = objectstore.get_session()
batchsize = 100
query = select( [records.c.record_id])
results = query.execute()
record_ids = [ row[0] for row in results.fetchall() ]
record_count = len(record_ids)
batches = [backup_ids[i:i+batchsize] for i in range(0, len
(backup_ids), batchsize)]
batchnum = 0
debug('')
for batch in batches:
batchnum += 1
debug('[ Batch %d ] (%d)' % (batchnum, len(batch)))
records = RecordLogMsg.mapper.select
(recordlog.c.record_id.in_(*batch))
for record in records:
debug(' %s' % (record.record_id))
session.expunge(record)
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your
job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?
cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
