On 05/17/2012 02:44 PM, Ceri Storey wrote:
Also, it's worth bearing in mind that heapy will only record objects
allocated and tracked by python, which will of course be a subset of
the total memory allocated to python by the os (which is what the
virtual size (VIRT) measures). I'd guess that if you were to take
heapy snapshots part way through creating the XML file, then you would
see a bigger allocation of objects.
I hope this is somehow useful,
It is. :) I was just about to post an update. I have isolated the
problem to be with SQLA+psycopg2 somewhere, because of the large number
of rows queried from the database at once. Switching to tempfile did
reduce memory from one large ElementTree by some 20%, and the rest is
caused by ORM'ed data itself. There are workarounds I'm pursuing on the
SQLA list which basically involves reading least amount of data possible
at once so that the request does not handle tens of thousands of fully
related SQLA objects at once, but in subsets.
For reference:
https://groups.google.com/d/msg/sqlalchemy/yAlrzMAH-Yk/FiNzFknjezMJ
.oO V Oo.
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/pylons-discuss?hl=en.
