Re: Why on CentOS, python consumes too much memory ?
I found that it was caused by not by python but by /usr/lib/locale/locale-archive, the same problem as that described in http://illiterat.livejournal.com/4615.html. William 在 2013年7月18日星期四UTC+8下午12时45分01秒，William Bai写道： Hi: Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M. On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M). And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python: import time time.sleep(200) I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/pid/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS 6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M. I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much. Best Regards William -- http://mail.python.org/mailman/listinfo/python-list
Re: Why on CentOS, python consumes too much memory ?
snowingbear at gmail.com writes: Hi: Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M. On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M). Take a look at http://www.selenic.com/smem/ for accurate measurement of actual memory consumption under Linux. Virtual memory size is generally useless for this purpose. Regards Antoine. -- http://mail.python.org/mailman/listinfo/python-list
Re: Why on CentOS, python consumes too much memory ?
On 07/18/2013 03:13 AM, William Bai wrote: I found that it was caused by not by python but by /usr/lib/locale/locale-archive, the same problem as that described in http://illiterat.livejournal.com/4615.html. Too funny. So in other words there isn't a problem at all. What you thought was RAM usage was really just a memory-mapped file. That's why Antoine said that using VSS to determine memory usage is not valid at all in determining memory footprint. VSS shows all kinds of things from shared libraries to locale archives that have zero impact on RAM usage. Every app will show at least the size of glibc in its VSS number. What is too much memory anyway? What do you mean by consume too much memory? Now if your script has a resource leak and is long-running, then that's a problem, but the solution is to fix your resource leak, not have the OS kill your app when it exceeds the RLIMIT. -- http://mail.python.org/mailman/listinfo/python-list
Why on CentOS, python consumes too much memory ?
Hi: Previously, we found that our python scripts consume too much memory. So I use python's resource module to restrict RLIMIT_AS's soft limit and hard limit to 200M. On my RHEL5.3(i386)+python2.6.2, it works OK. But on CentOS 6.2(x86_64)+python2.6.6, it reports memory error(exceeding 200M). And I tested with a very small script, and result is out of my expect, it still use too much memory on my CentOS 6.2 python: import time time.sleep(200) I use guppy and memory_profiler to see the memory usage and see that python objects just use about 6M memory both on RHEL5.3(i386)+python2.6 and CentOS 6.2(x86_64)+python2.6. But when I cat /proc/pid/status. I found that though VmRss is not very large on both machines. But the VmSize on CentOS 6.2(x86_64)+python2.6 is 140M-180M, while on my RHEL5.3+python2.6, the VmSize is just 6M. And I tested on CentOS 5.7(x86_64)+python2.4.3, the VmSize is 70M. I could understand that 64 bit machines will occupy more virtual memory than that on 32 bit, because the length of some types are not the same. But I don't know why they differs so greatly(6M to 180M), Or is this only caused by that python2.6 on CentOS 6.2's memory allocation is different from python's default one? Could you kindly give me some clues? Thank you very much. Best Regards William -- http://mail.python.org/mailman/listinfo/python-list
