yah, Ok i did expunge in a hurry so it was effectively a no-op.
there was also a lingering hard ref in the attributes module which
has been changing a lot lately. as it turns out, your example cleans
itself out anyway without the expunge since you arent changing the
loaded objects. i added a full unit test for this whole shebang
which loads batches of 100 at a time from a set of 25000, changes the
attributes, verifies that theres 100 units in the "dirty", "identity
map" and "managed attributes" lists on each pass and then does an
expunge for that batch, and the memory stays constant.
changeset 1296.
On Apr 19, 2006, at 12:33 PM, Peter L. Buschman wrote:
Hi Michael:
Thanks for taking the time.
In my example, the memory usage of python.exe started at about 26MB
and grew about 10MB with
each batch processed.
On your suggestion, I replaced the mapper's select() call with the
table's select() call like this:
records = recordlog.select(recordlog.c.record_id.in_
(*batch)).execute().fetchall()
After commenting-out the call to session.expunge(), I ran the
script again. Where the previous script
easily grew to 500+ MB in size, this one ran like lightning with
memory usage hovering around 25-26MB.
Granted, a simple ResultProxy has nowhere near the overhead of a
mapper, but it does seem to indicate
that the mapper isn't freeing-up memory the way I was using it.
I've also tried creating my engine with echo=True and running the
logged query against the database
outside of Python. Query analyzer returned quite quickly without
showing a significant memory increase
and neither did SQL Server.
Basically, what I want to do is to select from a mapper for each
set of records in a batch, make changes
to some of them, call objectstore.commit(), and then flush the
whole lot from memory before loading the
next batch.
FWIW, I am running this with SQL Server 2000, CPython 2.4.3, and
adodbapi.
--Peter
At 17:44 19.4.06, Michael Bayer wrote:
peter -
i think your approach is correct in theory, but i havent done a lot
of testing with "expunge" to insure that it cleans everything up as
its a recently added feature.
I would try to make sure it is indeed the mapper call that is growing
memory, such as by trying the below loop without using the mapper
call (maybe something else is doing it, like the plain SQL, or
something else). also try seeing if doing a session.clear() has
better results than expunge().
also my observations with CPython is that even under total memory-
cleaning conditions, the interpreter still can grow slightly.
ill take a look at this myself.
On Apr 19, 2006, at 11:03 AM, Peter L. Buschman wrote:
All:
I'm trying to process a large set of records in batches to control
memory usage, but after each batch,
the memory usage keeps going up. I suspect I am not freeing-up the
processed mapped objects correctly.
Can anyone here enlighten me as to what I'm doing wrong? Is there
another function I should be calling other
than session.expunge() once I have processed a record and am ready
to discard it? Out of the entire set,
only a small number will need to be updated so I can discard the
majority once I have determined that they
are not eligible to be changed.
The complete record set can be quite huge so I desperately need a
way to control Python's memory usage
while iterating through result sets. :(
Kindest regards,
Peter Buschman
session = objectstore.get_session()
batchsize = 100
query = select( [records.c.record_id])
results = query.execute()
record_ids = [ row[0] for row in results.fetchall() ]
record_count = len(record_ids)
batches = [backup_ids[i:i+batchsize] for i in range(0, len
(backup_ids), batchsize)]
batchnum = 0
debug('')
for batch in batches:
batchnum += 1
debug('[ Batch %d ] (%d)' % (batchnum, len(batch)))
records = RecordLogMsg.mapper.select
(recordlog.c.record_id.in_(*batch))
for record in records:
debug(' %s' % (record.record_id))
session.expunge(record)
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your
job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?
cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your
job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?
cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services,
security?
Get stuff done quickly with pre-integrated technology to make your
job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache
Geronimo
http://sel.as-us.falkag.net/sel?
cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
-------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security?
Get stuff done quickly with pre-integrated technology to make your job easier
Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
Sqlalchemy-users mailing list
Sqlalchemy-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/sqlalchemy-users
