RE: [Zope] Memory Leak Debugging

2006-02-28 Thread Jonathan Bowlas
Hi All,

A few weeks ago I posted this message to the list but we still haven't
resolved our memory leak, mainly because we have no idea where to look.

So having trawled around a few forums I decided to install the LeakFinder
product onto our development server and click around a little watching the
refcounts. Here I noticed that Products.Silva.Image.Image 
DateTime.DateTime.DateTime had the largest number of references. So
following the LeakFinder instructions I attempted to patch each of these
classes.

I started with the Products.Silva.Image.Image class and continued to click
around to return some results, unfortunately this never returned any results
at all so that wasn't much help. Next I patched the
DateTime.DateTime.DateTime class which returned a huge amount of information
on the results tab. So I'm now hoping someone could help me in deciphering
these results and explain what exactly should I be looking for here? And it
would also be useful to know why the Products.Sivla.Image.Image class
doesn't return any results at all.

Hope someone can help.

Jon

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of
Jonathan Bowlas
Sent: 15 February 2006 11:49
To: zope@zope.org
Subject: RE: [Zope] Memory Leak Debugging

Hi All,

We're having serious problems with performance running Silva CMS on Zope
2.7.8-final, python 2.3.5, linux2 set up on a machine with 2.5GB RAM. There
are currently only about 65 users and around 2000 documents in the CMS but
it seems to run for a week, then the RAM fills up and crashes so we have to
restart the server manually.

The Zope logs aren't much help as I enquired on the Zope list and their
response was we need more RAM. But surely 2.5GB is enough.

I've also checked Control_Panel/DebugInfo for any refcounts with significant
jumps and Products.Silva.Image.Image seems to be the culprit:
   11.32am  11.36am
Products.Silva.Image.Image     56203    58153      +1950
Acquisition.ImplicitAcquirerWrapper    44825    45151      +326

So we've tried to figure out which kind of request triggers these refcount
jumps by creating a separate instance with a very small ZODB Cache (so it
gets filled up soon), and clicked around a little, watching the refcounts
but we cannot seem to replicate it.

So basically this is a plea for help from an expert Zope debugger who can
assist us in finding the source of the problem so we can resolve it. Or
alternatively someone who can give us a set of instructions that we can
follow that will enable us to locate the problem, I've googled it and there
are no clear answers.

FYI we have the following products installed on our Zope instance.

Annotations 0.4.3
BTreeFolder2-1.0.1
DocFinderTab (0.5.2)
External Method-1-0-0
FileSystemSite 1.4.2
Formulator 1.10
Groups 0.4
IssueTrackerProduct (0.6.13)
LDAPUserFolder (2.3)
MIMETools
MailHost-1-3-0
OFSP-1-0-0
PageTemplates-1-4-0
ParsedXML 1.4
PlacelessTranslationService (1.0)
PluginIndexes
ProxyIndex 1.2
PythonScripts-2-0-0
Sessions
Silva 1.4.1
SilvaBlog 0.4.2
SilvaCustomAutoTOC 0.9
SilvaDocument 1.4.1
SilvaDocumentPatches 1.3b6-1
SilvaExtETHLDAP 0.7
SilvaExternalSources 0.10.3
SilvaMetadata 0.10
SilvaNewsNetwork 2.0
SilvaReferenceChecker 0.2
SilvaStylesheets 0.6.2
SilvaUCLAttributes 0.7 - (A UCL product*)
Silva UCL Gallery 0.1 - (A UCL product*)
SilvaUCLSubjectAudienceFilters 0.6 - (A UCL product*)
SilvaViews 0.11
SiteAccess-2-0-0
SiteErrorLog Sprout 0.6.2
StandardCacheManagers-1-1-0
TemporaryFolder
Transience
Tropos (1.9.12)
XMLWidgets-Pluggable 0.13 (CVS/UNRELEASED)
ZCTextIndex
ZCatalog-2-2-0
ZGadflyDA
ZMySQLDA 2.0.8
ZODBMountPoint
ZOracleDA
ZReST (1.1)
ZSQLMethods
Zope Tutorial 1.2
kupu 1.3.2 
z3locales 0.1

If you require any further information do not hesitate to ask.

Hope someone can help.

Jon Bowlas
University College London


___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


RE: [Zope] Memory Leak Debugging

2006-02-28 Thread Dieter Maurer
Jonathan Bowlas wrote at 2006-2-28 13:20 -:
 ...
I've also checked Control_Panel/DebugInfo for any refcounts with significant
jumps and Products.Silva.Image.Image seems to be the culprit:
   11.32am  11.36am
Products.Silva.Image.Image     56203    58153      +1950
Acquisition.ImplicitAcquirerWrapper    44825    45151      +326

Looks as it you would leak acquisition wrappers...


This can e.g. happen when you (or some extension product) store
acquisition wrappers in REQUEST outside of other or form.
Almost surely, other causes (e.g. caching acquisition wrappers)
are possible, too.

-- 
Dieter
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Memory Leak Debugging

2006-02-15 Thread Peter Bengtsson
Hi,
I'm NOT an expert and I hope that someone else who is an expert gives
you better advise but here's a start...
Do you have any recursive loops with lists that doesn't reset the
list? Perhaps something like this:

def findAllImages(self, placetolook, append2):
for o in placetolook.objectValues():
if o.meta_type == 'Silva Image':
append2.append(o)
elif o.isPrincipiaFolderish:
findAllImages(o, append2)
or something like that.

I had a memory leak and the refcounts pointed to the Photo class.
Turns out that I did a bizarre loop over these without resetting it
properly after. It was a bitch to debug and it was a long time ago
when Zope was called zope 2.3.

That's all I've got to offer. Scan your code for places where you
create lists by loops on the ZODB. 2000 objects is enough if they are
piled up and not freed until a server restart. Heck, if they're big,
200 is enough.

On 2/15/06, Jonathan Bowlas [EMAIL PROTECTED] wrote:
 Hi All,

 We're having serious problems with performance running Silva CMS on Zope
 2.7.8-final, python 2.3.5, linux2 set up on a machine with 2.5GB RAM. There
 are currently only about 65 users and around 2000 documents in the CMS but
 it seems to run for a week, then the RAM fills up and crashes so we have to
 restart the server manually.

 The Zope logs aren't much help as I enquired on the Zope list and their
 response was we need more RAM. But surely 2.5GB is enough.

 I've also checked Control_Panel/DebugInfo for any refcounts with significant
 jumps and Products.Silva.Image.Image seems to be the culprit:
11.32am  11.36am
 Products.Silva.Image.Image 5620358153  +1950
 Acquisition.ImplicitAcquirerWrapper4482545151  +326

 So we've tried to figure out which kind of request triggers these refcount
 jumps by creating a separate instance with a very small ZODB Cache (so it
 gets filled up soon), and clicked around a little, watching the refcounts
 but we cannot seem to replicate it.

 So basically this is a plea for help from an expert Zope debugger who can
 assist us in finding the source of the problem so we can resolve it. Or
 alternatively someone who can give us a set of instructions that we can
 follow that will enable us to locate the problem, I've googled it and there
 are no clear answers.

 FYI we have the following products installed on our Zope instance.

 Annotations 0.4.3
 BTreeFolder2-1.0.1
 DocFinderTab (0.5.2)
 External Method-1-0-0
 FileSystemSite 1.4.2
 Formulator 1.10
 Groups 0.4
 IssueTrackerProduct (0.6.13)
 LDAPUserFolder (2.3)
 MIMETools
 MailHost-1-3-0
 OFSP-1-0-0
 PageTemplates-1-4-0
 ParsedXML 1.4
 PlacelessTranslationService (1.0)
 PluginIndexes
 ProxyIndex 1.2
 PythonScripts-2-0-0
 Sessions
 Silva 1.4.1
 SilvaBlog 0.4.2
 SilvaCustomAutoTOC 0.9
 SilvaDocument 1.4.1
 SilvaDocumentPatches 1.3b6-1
 SilvaExtETHLDAP 0.7
 SilvaExternalSources 0.10.3
 SilvaMetadata 0.10
 SilvaNewsNetwork 2.0
 SilvaReferenceChecker 0.2
 SilvaStylesheets 0.6.2
 SilvaUCLAttributes 0.7 - (A UCL product*)
 Silva UCL Gallery 0.1 - (A UCL product*)
 SilvaUCLSubjectAudienceFilters 0.6 - (A UCL product*)
 SilvaViews 0.11
 SiteAccess-2-0-0
 SiteErrorLog
 Sprout 0.6.2
 StandardCacheManagers-1-1-0
 TemporaryFolder
 Transience
 Tropos (1.9.12)
 XMLWidgets-Pluggable 0.13 (CVS/UNRELEASED)
 ZCTextIndex
 ZCatalog-2-2-0
 ZGadflyDA
 ZMySQLDA 2.0.8
 ZODBMountPoint
 ZOracleDA
 ZReST (1.1)
 ZSQLMethods
 Zope Tutorial 1.2
 kupu 1.3.2
 z3locales 0.1

 If you require any further information do not hesitate to ask.

 Hope someone can help.

 Jon Bowlas
 University College London

 ___
 Zope maillist  -  Zope@zope.org
 http://mail.zope.org/mailman/listinfo/zope
 **   No cross posts or HTML encoding!  **
 (Related lists -
  http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope-dev )



--
Peter Bengtsson,
work www.fry-it.com
home www.peterbe.com
hobby www.issuetrackerproduct.com
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Memory Leak Debugging

2006-02-15 Thread Andrew Milton
+---[ Jonathan Bowlas ]--
| Hi All,
| 
| We're having serious problems with performance running Silva CMS on Zope
| 2.7.8-final, python 2.3.5, linux2 set up on a machine with 2.5GB RAM. There
| are currently only about 65 users and around 2000 documents in the CMS but
| it seems to run for a week, then the RAM fills up and crashes so we have to
| restart the server manually.
| 
| The Zope logs aren't much help as I enquired on the Zope list and their
| response was we need more RAM. But surely 2.5GB is enough.
| 
| I've also checked Control_Panel/DebugInfo for any refcounts with significant
| jumps and Products.Silva.Image.Image seems to be the culprit:
|11.32am  11.36am
| Products.Silva.Image.Image   5620358153  +1950
| Acquisition.ImplicitAcquirerWrapper  4482545151  +326

Is there a possibility something is webcrawling your site?

-- 
Andrew Milton
[EMAIL PROTECTED]
___
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )