Hi Maarten, Your link http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete- a-large-object.htm seems to suggest that even though the interpreter does not release memory back to the OS, it can be re-used by the interpreter. If this was our problem, I'd expect the memory to be set by the highest usage, as opposed to it constantly leaking: in my case, the load is virtually constant, but the memory still leaks over time.
The environment is Linux 2.6.24 x86-64, the extensions used are MySQLdb, pyCrypto (latest stable releases for both). > -----Original Message----- > From: twisted-python-boun...@twistedmatrix.com [mailto:twisted-python- > boun...@twistedmatrix.com] On Behalf Of Maarten ter Huurne > Sent: Monday, February 22, 2010 6:24 PM > To: Twisted general discussion > Subject: Re: [Twisted-Python] debugging a memory leak > > On Tuesday 23 February 2010, Alec Matusis wrote: > > > When I start the process, both python object sizes and their counts rise > > proportionally to the numbers of reconnected clients, and then they > > stabilize after all clients have reconnected. > > At that moment, the "external" RSS process size is about 260MB. The > > "internal size" of all python objects reported by Heapy is about 150MB. > > After two days, the internal sizes/counts stay the same, but the external > > size grows to 1500MB. > > > > Python object counts/total sizes are measured from the manhole. > > Is this sufficient to conclude that this is a C memory leak in one of the > > external modules or in the Python interpreter itself? > > In general, there are other reasons why heap size and RSS size do not match: > 1. pages are empty but not returned to the OS > 2. pages cannot be returned to the OS because they are not completely empty > > It seems Python has different allocators for small and large objects: > http://www.mail-archive.com/python-l...@python.org/msg256116.html > http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i- > delete- > a-large-object.htm > > Assuming Python uses malloc for all its allocations (does it?), it is the > malloc implementation that determines whether empty pages are returned to > the OS. Under Linux with glibc (your system?), empty pages are returned, so > there reason 1 does not apply. > > Depending on the allocation behaviour of Python, the pages may not be > empty > though, so reason 2 is a likely suspect. > > Python extensions written in C could also leak or fragment memory. Are you > using any extensions that are not pure Python? > > Bye, > Maarten > > _______________________________________________ > Twisted-Python mailing list > Twisted-Python@twistedmatrix.com > http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
