Re: [Zope] A memory leak in Zope, PageTemplateFile and macros?
Marco Bizzarri wrote at 2005-5-6 22:35 +0200:
> ...
>However, if you access start_html, which uses the standard_template_html
>as a macro, you can see an increase.
I saw a small one after thousands of requests.
> ...
>You're mentioning that these could be normal causes,
>but what are these causes?
Small memory consumption increases can be caused by an effect
known as memory fragmentation. The "C" runtime library (and Python
on top of it) does not support memory compaction. Therefore, free memory
areas can only be joined together if they are adjacent (and even then
memory management may not do it). As a consequence, the free memory
tends to be fragmented in a large number of small blocks.
> ...
>I've tried this both on 2.7.3 and 2.7.6, and in both cases I'm
>experiencing the problem. Do you have any suggestion on what I could
>possibly investigate?
The usual approach in investigating memory leaks works as follows:
* you flush the ZODB caches
("Control_Panel --> Database Management --> --> Flush cache
--> "minimize").
* you visit "Control_Panel --> Debug information" and
"update snapshot"
* you perform your test
* you flush the ZODB caches
* you visit "Control_Panel" --> "Debug information"
and analyse the changes in reference counts (in the middle
of the page).
This allows you to recognize leaks at the level of instances (of classes).
The "LeakFinder" product helps you with the above steps.
I did this for your case and could not determine leaking instances.
When instances of static types (e.g. strings, integers, dictionaries, ...)
leak, they cannot be detected in this way. Neither can
leaked caused by fragmented memory.
Python can be build in a special way to analyse leaks of
any Python objects. Note, however, that all "C" extensions
(especially Zope) must be then build in a consistent way.
Tracking down such leaks is even more expensive that
those involving standard Zope objects.
There are tools around to analyse leaks at the "C" level.
I had used the commercial product "purify" once (a long time ago).
Andreas (Jung) told me that some open source project (I think
it was KDE) has built a similar tool.
--
Dieter
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] A memory leak in Zope, PageTemplateFile and macros?
Dieter Maurer wrote: Marco Bizzarri wrote at 2005-5-5 18:07 +0200: While debugging an application of us I suspected we had a memory leak, so I started with the usual divide et impera approach, trying to figure where exactly was the problem. I tried to reproduce your problem and I used 1.000 requests per round. Memory does not grow at all when your templates are put into the ZODB (as "ZopePageTemplates"). After tens of thousands requests, I could observe a small increase in the memory footprint when I used your full example. If there is a problem, then it is probably with "PageTemplateFile". However, the memory increase was small enough that it could have completely normal causes... I did two different tests: if you try to access to the standard_template_html, you won't see the memory footprint rise. However, if you access start_html, which uses the standard_template_html as a macro, you can see an increase. In both cases I'm speaking about PageTemplateFile (which we are using inside our project). If you use ab with let's say 10.000 requests, you can see a constant increase in memory. You're mentioning that these could be normal causes, but what are these causes? I mean, I started this investigation in order to discover any leak in our project, but it is difficult to do if I'm unable to remove all other possible causes. Thus, I cannot reproduce your problem. I tried with Zope 2.7.2 (+ patches) I've tried this both on 2.7.3 and 2.7.6, and in both cases I'm experiencing the problem. Do you have any suggestion on what I could possibly investigate? Thanks for your answer. Regards Marco ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] A memory leak in Zope, PageTemplateFile and macros?
Marco Bizzarri wrote at 2005-5-5 18:07 +0200: >While debugging an application of us I suspected we had a memory leak, >so I started with the usual divide et impera approach, trying to figure >where exactly was the problem. I tried to reproduce your problem and I used 1.000 requests per round. Memory does not grow at all when your templates are put into the ZODB (as "ZopePageTemplates"). After tens of thousands requests, I could observe a small increase in the memory footprint when I used your full example. If there is a problem, then it is probably with "PageTemplateFile". However, the memory increase was small enough that it could have completely normal causes... Thus, I cannot reproduce your problem. I tried with Zope 2.7.2 (+ patches). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] A memory leak in Zope, PageTemplateFile and macros?
Hi all. While debugging an application of us I suspected we had a memory leak, so I started with the usual divide et impera approach, trying to figure where exactly was the problem. I therefore started removing all code from inside the main templates in our system, so that I could figure where the problem was. The surprise was that even if I removed all the code, still I was able to see the memory footprint of the zope server raise as new requests arrived. I'm using ab from Apache project to test. I've devised a very simple product which shows the problem, which I include. I tested this with python 2.3.4 and Zope 2.7.6 Test string was: ab2 -A admin:admin -n 100 http://localhost:8080/LeakPT1/start_html Regards Marco LeakPT1.tar.gz Description: application/gzip ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
