On 05/06/2012 10:11 PM, Ben Bangert wrote:
I found this presentation quite fascinating regarding how Python handles memory: http://revista.python.org.ar/2/en/html/memory-fragmentation.html
Oh yeah, I've seen that presentation before, it is very interesting.
Which version of Python? You might try running some of the memory profiling tools, perhaps you still have a reference to the XML structure you weren't aware of, and that's why it wasn't freed up. Alternatively, if that's really bad, consider generating the XML with generators so that you never have the entire thing in RAM at once.
It's Python 2.6.6, default on CentOS 6.2, 32-bit. I'll check with a profiler to see if anything lingers in the lxml namespace because the objects I'm creating directly live only within the view handler.
I'm not sure I can use generators directly on XML because the XML is basically a huge list of dictionaries taken from database, processed and transformed to adequate XML data, and it happens occasionally that some exception occurs that MUST prevent generation of the XML, ie. return http error response, rather than 200 and broken XML. The only alternative is output to a temp file and then use generators to feed that file back to client.
With that, I assume I can put a tempfile.TemporaryFile object in env and expect it to be closed/destroyed AFTER the request is complete?
There's a few interesting alternatives to generating large XML files without the memory hit here: http://stackoverflow.com/questions/3049188/generating-very-large-xml-files-in-python
Interesting, thanks, and that's just a variant of the tempfile approach. .oO V Oo. -- You received this message because you are subscribed to the Google Groups "pylons-discuss" group. To post to this group, send email to [email protected]. To unsubscribe from this group, send email to [email protected]. For more options, visit this group at http://groups.google.com/group/pylons-discuss?hl=en.
