On 05/05/2012 07:42 PM, Vlad K. wrote:
Hi all.
As I understand it Python won't release internally released memory back
to OS. So it happens in my Pyramid app that has to process/construct
rather largish XML files occasionally, the memory consumption jumps to
several hundred MB and stays there for good or until I recycle the
processes (uwsgi reload). This is pretty bad in my case because I'm
forced to have a rather large memory headroom on the server just to
handle those peaks which comprise less than 1% of total requests
throughout the day. Hell, it's way less than 1%. A single request for
such XML file takes anywhere from 1 to up to 10 seconds on this
particular VPS "hardware", and there are maybe a few dozen such requests
throughout the day (out of several thousand daily requests). So what
should really be a transient memory consumption peak lasting up to 10
seconds becomes permanent memory requirement (or until the uwsgi
processes are reloaded).
Now I know I can set max-requests for the uwsgi processes and it will
recycle automatically, but is there another solution? I find this
reluctance of Python to return memory to OS very annoying, right next to
the GIL.
Commenting on how Python does or doesn't return memory to the OS is
above my current pay grade. Because I don't understand it, and have no
competence to try to "fix" it, I have to work around it. I usually try
to do that by writing code doesn't ask for hundreds of megabytes all in
one shot from the OS.
For example, maybe you can construct the XML in portions instead of
constructing a huge collection of strings that all reside in memory at once.
If there's a memory leak somewhere outside of my control, what I usually
do to work around it is to use supervisor + memmon to automatically
restart my processes when they use more than some amount of memory:
http://www.plope.com/Members/chrism/memmon_sample
This isn't immediately useful if your processes are spawned by uwsgi,
but might be useful if you choose to use a different frontend server
like Apache or nginx to proxy to a number of backend supervisor-managed
Pyramid processes.
- C
--
You received this message because you are subscribed to the Google Groups
"pylons-discuss" group.
To post to this group, send email to [email protected].
To unsubscribe from this group, send email to
[email protected].
For more options, visit this group at
http://groups.google.com/group/pylons-discuss?hl=en.
