Re: Detecting memory leaks on apache, mod_python
On Dec 23, 2:47 am, Steven D'Aprano [EMAIL PROTECTED] cybersource.com.au wrote: On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote: I've never encountered such items supported by the language. OS specific extensions MIGHT supply it... Picky picky... but of course you are right. When I said that programming languages I have used before had facilities to measure memory usage, I meant that the *implementation* had those facilities rather than the language itself. yes, that's what this thread is about. I'm really just looking for a low-level python (memory) profiling tool, and thus i'm asking in c.l.p. for experiences, mainly in context of an apache mod_python environment. But seeing the responses within this thread, it looks like there's no such tool available. . http://case.lazaridis.com/wiki/PythonAudit -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Sat, 22 Dec 2007 08:09:50 +0100, Fredrik Lundh wrote: Steven D'Aprano wrote: Not me. You're quite knew to this internet thing, aren't you? ;-) :-D So... how do you measure memory usage in Python? Every programming language I've used before (not a huge range, I'll admit) had *some* sort of facility to measure memory usage, typically things like: * how much memory is free in the stack? * how much memory is free in the heap? * how big a block does this pointer point to? * how much memory does this record/struct/object/string use? And what languages would that be? I cannot think of a single modern language that does any of that. Including low-level stuff like C/C++. I didn't actually say they were *modern* languages. E.g. THINK Pascal for Apple Mac, circa 1990. And things like how much memory is free in the heap isn't even a meaningful concept on a modern machine, thanks to the wonders of virtual memory (especially the overcommitting kind). Maybe the memory model used in modern multi-tasking virtual-memory PCs makes the concept of free memory obsolete. Although if so, somebody should have a quiet word with the author of the Linux free command: $ free total used free sharedbuffers cached Mem: 1002524 988736 13788 0 7044 98916 -/+ buffers/cache: 882776 119748 Swap: 42410803939736 301344 (Admittedly that's system-wide memory usage, rather than for a single process.) For Python, standard process monitoring tools (combined with a basic understanding of how dynamic memory allocation works on modern platforms) are usually sufficient to get a good view of an application's memory usage patterns. Just run the program under a few different scenarios, and see what it does. Are you saying that Python programs can't monitor their own memory use? I'm happy to accept that free memory is not a meaningful concept for a process in a modern system. That makes sense. But surely it is reasonable for a process to have an idea of how much memory it has actually used. Yes? No? If the memory use looks suspicious, use standard debugging techniques to locate the problematic area, and standard benchmarking techniques to look for unexpected blowups and leaks. What sort of standard debugging techniques work in the absence of any way (that I know of) to measure memory usage? In Python, standard debugging techniques usually start with the print statement, but one can't do anything like this: # problematic area of code last = memory() for i in xrange(100): x = foo() if memory() = last: print memory use increased, memory() So what are you suggesting is standard? For really hard problems, use the gc module, instrumenting memory allocation libraries (e.g. dmalloc), or C/C++-level debuggers. I'm not so much concerned about the really hard problems as I am about the really easy ones. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
Steven D'Aprano wrote: Not me. You're quite knew to this internet thing, aren't you? ;-) :-D (hmm. why is that whenever you make some silly last-second addition to a post, you end up making a stupid typo?) And things like how much memory is free in the heap isn't even a meaningful concept on a modern machine, thanks to the wonders of virtual memory (especially the overcommitting kind). Maybe the memory model used in modern multi-tasking virtual-memory PCs makes the concept of free memory obsolete. Although if so, somebody should have a quiet word with the author of the Linux free command: $ free total used free sharedbuffers cached Mem: 1002524 988736 13788 0 7044 98916 -/+ buffers/cache: 882776 119748 Swap: 42410803939736 301344 (Admittedly that's system-wide memory usage, rather than for a single process.) the problem isn't that you can get a number, it's that the number you get has no *direct* correlation to the amount of memory your process can actually allocate -- or at least allocate without causing excessive swapping or triggering the OOM killer or otherwise annoying all the other processes on the machine. For Python, standard process monitoring tools (combined with a basic understanding of how dynamic memory allocation works on modern platforms) are usually sufficient to get a good view of an application's memory usage patterns. Just run the program under a few different scenarios, and see what it does. Are you saying that Python programs can't monitor their own memory use? I'm happy to accept that free memory is not a meaningful concept for a process in a modern system. That makes sense. But surely it is reasonable for a process to have an idea of how much memory it has actually used. Yes? No? unless you do utterly trivial things, a process allocates memory from a lot of different resource pools (i/o buffers, virtual memory buffers, network buffers, graphics resources, etc). there's simply no way for the process itself to know how much memory it uses. if you want to know, ask the operating system. If the memory use looks suspicious, use standard debugging techniques to locate the problematic area, and standard benchmarking techniques to look for unexpected blowups and leaks. What sort of standard debugging techniques work in the absence of any way (that I know of) to measure memory usage? as I said, use standard process monitoring tools (i.e. ps and top etc on Unix, the task manager on Windows) takes you a long way. In Python, standard debugging techniques usually start with the print statement but one can't do anything like this: # problematic area of code last = memory() for i in xrange(100): x = foo() if memory() = last: print memory use increased, memory() it doesn't have to be harder than this: # problematic area of code raw_input(before: check memory here) for i in xrange(100): x = foo() raw_input(after: check memory here) on unix, you can also do e.g. os.system(ps ux) or os.system(ps -F -p %d % os.getpid()) or some variation thereof. So what are you suggesting is standard? running individual modules/classes/functions under test harnesses, so you can analyze their behaviour under well-known conditions. /F -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On 22 Δεκ, 09:09, Fredrik Lundh [EMAIL PROTECTED] wrote: [...] For Python, standard process monitoring tools (combined with a basic understanding of how dynamic memory allocation works on modern platforms) are usually sufficient to get a good view of an application's memory usage patterns. [...] which standard process monitoring tools are those? Aren't there any python specific tools (e.g. which list the python modules which use the memory)? . http://case.lazaridis.com/wiki/PythonAudit -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Sat, 22 Dec 2007 13:05:23 -0800, Dennis Lee Bieber wrote: I've never encountered such items supported by the language. OS specific extensions MIGHT supply it... Picky picky... but of course you are right. When I said that programming languages I have used before had facilities to measure memory usage, I meant that the *implementation* had those facilities rather than the language itself. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Dec 19, 5:40 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: On Dec 17, 8:41 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: How to detect memory leaks of python programms, which run in an environment like this: * Suse Linux 9.3 * Apache * mod_python The problem occoured after some updates on the infrastructure. It's most possibly caused by trac and it's dependencies, but several components of the OS where updated, too. Any nice tools which play with the above constellation? Thank's for any hints! No tool available to detect memory leaks for python applications? So, anyone who hit's on this thread via a search will think a) that there's really no memory leak detection for python b) that this community is not very helpful context: http://dev.lazaridis.com/base/ticket/148 -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Dec 21, 7:42 pm, Ilias Lazaridis [EMAIL PROTECTED] wrote: On Dec 19, 5:40 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: On Dec 17, 8:41 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: How to detect memory leaks of python programms, which run in an environment like this: * Suse Linux 9.3 * Apache *mod_python The problem occoured after some updates on the infrastructure. It's most possibly caused by trac and it's dependencies, but several components of the OS where updated, too. Any nice tools which play with the above constellation? Thank's for any hints! No tool available to detect memory leaks for python applications? So, anyone who hit's on this thread via a search will think a) that there's really no memory leak detection for python b) that this community is not very helpful Comments like (b) will not help your chances one bit in respect of getting an answer from anyone. Maybe you should read: http://www.catb.org/~esr/faqs/smart-questions.html Two things to note in here. First, choose your forum appropriately and secondly show some courtesy rather than making accusations against the community if no one answers. If you want to see perhaps how you might be viewed by the open source community when you make such a comment, perhaps also watch: http://video.google.com/videoplay?docid=-4216011961522818645 Now, since you think this is a Trac problem, why don't you go ask on the Trac mailing list. http://groups.google.com/group/trac-users?lnk=li Even a search of that forum will most likely yield previous discussions about growing memory use of Trac as a result of things like Python wrappers for Subversion or certain database adapters. It may be a case of using a different version, or in some cases configuration of your hosting environment, if using Apache, to recycle Apache child processes after a set number of requests so as to restore process sizes back to a low level. So, do some research first in the correct places and then perhaps ask any additional questions in the correct place also. Graham -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Dec 21, 3:21 pm, Diez B. Roggisch [EMAIL PROTECTED] wrote: [...] Please get serious, Mr.! (and avoid further off-topics) . -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
So, anyone who hit's on this thread via a search will think a) that there's really no memory leak detection for python b) that this community is not very helpful c) That finally people in this forum are smart enough to detect your flamebait refuse to comment on it, Ilias... Comments like (b) will not help your chances one bit in respect of getting an answer from anyone. Maybe you should read: http://www.catb.org/~esr/faqs/smart-questions.html Two things to note in here. First, choose your forum appropriately and secondly show some courtesy rather than making accusations against the community if no one answers. If you want to see perhaps how you might be viewed by the open source community when you make such a comment, perhaps also watch: http://video.google.com/videoplay?docid=-4216011961522818645 Now, since you think this is a Trac problem, why don't you go ask on the Trac mailing list. http://groups.google.com/group/trac-users?lnk=li Even a search of that forum will most likely yield previous discussions about growing memory use of Trac as a result of things like Python wrappers for Subversion or certain database adapters. It may be a case of using a different version, or in some cases configuration of your hosting environment, if using Apache, to recycle Apache child processes after a set number of requests so as to restore process sizes back to a low level. So, do some research first in the correct places and then perhaps ask any additional questions in the correct place also. Graham, I'm not sure how long you have been around here - but googling for Ilias in general and specificly in this group will reveal that he often posts spiteful inflammatory messages and has a tendency to get cross with open source projects. It's debatable if he can be called a troll - but he surely is a nuisance best let alone, so he doesn't gain traction. Diez -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Dec 21, 12:25 pm, Graham Dumpleton [EMAIL PROTECTED] wrote: On Dec 21, 7:42 pm, Ilias Lazaridis [EMAIL PROTECTED] wrote: On Dec 19, 5:40 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: On Dec 17, 8:41 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: How to detect memory leaks of python programms, which run in an environment like this: * Suse Linux 9.3 * Apache *mod_python The problem occoured after some updates on the infrastructure. It's most possibly caused by trac and it's dependencies, but several components of the OS where updated, too. Any nice tools which play with the above constellation? Thank's for any hints! No tool available to detect memory leaks for python applications? So, anyone who hit's on this thread via a search will think a) that there's really no memory leak detection for python b) that this community is not very helpful Comments like (b) will not help your chances one bit in respect of getting an answer from anyone. [...] I'm really just looking for a low-level python (memory) profiling tool, and thus i'm asking in c.l.p. for experiences, mainly in context of an apache mod_python environment. As to Open Source etc.: http://case.lazaridis.com/wiki/PythonAudit http://case.lazaridis.com/wiki/CoreLiveEval Than you for your reply. . -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Fri, 21 Dec 2007 14:21:27 +0100, Diez B. Roggisch wrote: So, anyone who hit's on this thread via a search will think a) that there's really no memory leak detection for python b) that this community is not very helpful c) That finally people in this forum are smart enough to detect your flamebait refuse to comment on it, Ilias... Not me. I didn't (still don't actually) recognise Ilias' name. I was actually thinking: (d) Python is so great that there can never be any memory leaks! but I guess that's probably wishful thinking. So... how do you measure memory usage in Python? Every programming language I've used before (not a huge range, I'll admit) had *some* sort of facility to measure memory usage, typically things like: * how much memory is free in the stack? * how much memory is free in the heap? * how big a block does this pointer point to? * how much memory does this record/struct/object/string use? Unless I've missed something, I don't believe Python does *any* of that. It could be quite useful. The timeit module is good for measuring the time taken for code to run, but as near as I can tell, there's no way to optimize memory use except prematurely. -- Steven. -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
Steven D'Aprano wrote: Not me. You're quite knew to this internet thing, aren't you? ;-) So... how do you measure memory usage in Python? Every programming language I've used before (not a huge range, I'll admit) had *some* sort of facility to measure memory usage, typically things like: * how much memory is free in the stack? * how much memory is free in the heap? * how big a block does this pointer point to? * how much memory does this record/struct/object/string use? And what languages would that be? I cannot think of a single modern language that does any of that. Including low-level stuff like C/C++. And things like how much memory is free in the heap isn't even a meaningful concept on a modern machine, thanks to the wonders of virtual memory (especially the overcommitting kind). For Python, standard process monitoring tools (combined with a basic understanding of how dynamic memory allocation works on modern platforms) are usually sufficient to get a good view of an application's memory usage patterns. Just run the program under a few different scenarios, and see what it does. If the memory use looks suspicious, use standard debugging techniques to locate the problematic area, and standard benchmarking techniques to look for unexpected blowups and leaks. For really hard problems, use the gc module, instrumenting memory allocation libraries (e.g. dmalloc), or C/C++-level debuggers. /F -- http://mail.python.org/mailman/listinfo/python-list
Re: Detecting memory leaks on apache, mod_python
On Dec 17, 8:41 am, Ilias Lazaridis [EMAIL PROTECTED] wrote: How to detect memory leaks of python programms, which run in an environment like this: * Suse Linux 9.3 * Apache * mod_python The problem occoured after some updates on the infrastructure. It's most possibly caused by trac and it's dependencies, but several components of the OS where updated, too. Any nice tools which play with the above constellation? Thank's for any hints! No tool available to detect memory leaks for python applications? context: http://dev.lazaridis.com/base/ticket/148 -- http://mail.python.org/mailman/listinfo/python-list
Detecting memory leaks on apache, mod_python
How to detect memory leaks of python programms, which run in an environment like this: * Suse Linux 9.3 * Apache * mod_python The problem occoured after some updates on the infrastructure. It's most possibly caused by trac and it's dependencies, but several components of the OS where updated, too. Any nice tools which play with the above constellation? Thank's for any hints! context: http://dev.lazaridis.com/base/ticket/148 -- http://mail.python.org/mailman/listinfo/python-list
