On 5/5/12 4:42 PM, Vlad K. wrote: > As I understand it Python won't release internally released memory back > to OS.
This is actually no longer the case, I believe this behavior was fixed in Python 2.6 or so. If you're curious about Python memory allocation, there's also another interesting phenomenon in Python with memory fragmentation. I found this presentation quite fascinating regarding how Python handles memory: http://revista.python.org.ar/2/en/html/memory-fragmentation.html > So it happens in my Pyramid app that has to process/construct > rather largish XML files occasionally, the memory consumption jumps to > several hundred MB and stays there for good or until I recycle the > processes (uwsgi reload). Which version of Python? You might try running some of the memory profiling tools, perhaps you still have a reference to the XML structure you weren't aware of, and that's why it wasn't freed up. Alternatively, if that's really bad, consider generating the XML with generators so that you never have the entire thing in RAM at once. > This is pretty bad in my case because I'm > forced to have a rather large memory headroom on the server just to > handle those peaks which comprise less than 1% of total requests > throughout the day. Hell, it's way less than 1%. A single request for > such XML file takes anywhere from 1 to up to 10 seconds on this > particular VPS "hardware", and there are maybe a few dozen such requests > throughout the day (out of several thousand daily requests). So what > should really be a transient memory consumption peak lasting up to 10 > seconds becomes permanent memory requirement (or until the uwsgi > processes are reloaded). > > Now I know I can set max-requests for the uwsgi processes and it will > recycle automatically, but is there another solution? I find this > reluctance of Python to return memory to OS very annoying, right next to > the GIL. There's a few interesting alternatives to generating large XML files without the memory hit here: http://stackoverflow.com/questions/3049188/generating-very-large-xml-files-in-python -- Ben Bangert (ben@ || http://) groovie.org
signature.asc
Description: OpenPGP digital signature
