Hi Assuming that if memory not released to the OS can be reused by the interpreter because of a suballocation system used in the interpreter should eventually lead to a leveling out of the overall memory usage over time, that's what I observe with our processes (sitting at several 100 MB per process). We are using external C libraries which do lots of malloc/free and one of the bigger sources of pain is indeed to bring such a library to a point where its clean not only by freeing all memory allocated in every circumstance but also Python refcounting wise. I usually go thru all the motions to build up a complete debug chain for all modules involved in a project and write a test bed to proof clean and proper implementation.
So if your using C/C++ based modules in your project I would mark them as highly suspicious to be responsible for leaks until proven otherwise. Not to bother you with numbers but I usually allocate about 30% of overall project time to bring a server into a production ready state, meaning uptimes of months/years, no fishy feelings, no performance oscillations, predictable caving and recuperating when overloaded, just all the things you have to tick to sign off a project as completed, meaning you don't have to do daily 'tire kicking' maintenance and periodic reboots. Werner Alec Matusis wrote: > Hi Maarten, > > Your link > http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete- > a-large-object.htm > seems to suggest that even though the interpreter does not release memory > back to the OS, it can be re-used by the interpreter. > If this was our problem, I'd expect the memory to be set by the highest > usage, as opposed to it constantly leaking: in my case, the load is > virtually constant, but the memory still leaks over time. > > The environment is Linux 2.6.24 x86-64, the extensions used are MySQLdb, > pyCrypto (latest stable releases for both). > >> -----Original Message----- >> From: twisted-python-boun...@twistedmatrix.com [mailto:twisted-python- >> boun...@twistedmatrix.com] On Behalf Of Maarten ter Huurne >> Sent: Monday, February 22, 2010 6:24 PM >> To: Twisted general discussion >> Subject: Re: [Twisted-Python] debugging a memory leak >> >> On Tuesday 23 February 2010, Alec Matusis wrote: >> >>> When I start the process, both python object sizes and their counts rise >>> proportionally to the numbers of reconnected clients, and then they >>> stabilize after all clients have reconnected. >>> At that moment, the "external" RSS process size is about 260MB. The >>> "internal size" of all python objects reported by Heapy is about 150MB. >>> After two days, the internal sizes/counts stay the same, but the > external >>> size grows to 1500MB. >>> >>> Python object counts/total sizes are measured from the manhole. >>> Is this sufficient to conclude that this is a C memory leak in one of > the >>> external modules or in the Python interpreter itself? >> In general, there are other reasons why heap size and RSS size do not > match: >> 1. pages are empty but not returned to the OS >> 2. pages cannot be returned to the OS because they are not completely > empty >> It seems Python has different allocators for small and large objects: >> http://www.mail-archive.com/python-l...@python.org/msg256116.html >> http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i- >> delete- >> a-large-object.htm >> >> Assuming Python uses malloc for all its allocations (does it?), it is the >> malloc implementation that determines whether empty pages are returned to >> the OS. Under Linux with glibc (your system?), empty pages are returned, > so >> there reason 1 does not apply. >> >> Depending on the allocation behaviour of Python, the pages may not be >> empty >> though, so reason 2 is a likely suspect. >> >> Python extensions written in C could also leak or fragment memory. Are you >> using any extensions that are not pure Python? >> >> Bye, >> Maarten >> >> _______________________________________________ >> Twisted-Python mailing list >> Twisted-Python@twistedmatrix.com >> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python > > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
