[Zope-dev] Memory Leak, and Changing Severity of a Bug

2005-11-22 Thread Dan Pozmanter



I'd like to upgrade 
bug 1950 to critical.
Is there an easy way 
to do this?
___

Daniel PozmanterSiteworx, 
Inc.

"Festina Lente" - Gaius 
Julius

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Memory Leak Question

2005-11-07 Thread Dan Pozmanter
Consider it reported then.
(Sorry for the delay, I was out of the office for most of last week).

It looks like the problem is in Wrapper_findattr.

Any thoughts?

___
 
Daniel Pozmanter
Siteworx, Inc.
 
Festina Lente - Gaius Julius

-Original Message-
From: Chris McDonough [mailto:[EMAIL PROTECTED] 
Sent: Monday, October 31, 2005 10:33 PM
To: Dan Pozmanter
Cc: zope-dev@zope.org
Subject: Re: [Zope-dev] Memory Leak Question

Nobody has yet reported a memory leak symptom against any particular
piece of Zope 2.8.X yet, AFAIK.

On Oct 31, 2005, at 6:34 PM, Dan Pozmanter wrote:

 Ahoy,

I was poking around in _Acquisition.c, and noticed my memory usage 
 spiked.
 Returning to the vanilla code (2.8.2-final), I noticed the same spike 
 (just less pronounced).
 (If you refresh the page a few gazillion times this pops up).

 I would not categorize this as an actual bug report (I'd like to test 
 more rigorously with a script to verify a leak), but after reading the

 note on Changes for 2.8.x in the to do section, I was curious:
 Is this a known issue?

 If so, has there been any headway, and can I be of any assistance in 
 resolving it?

 ___

 Daniel Pozmanter
 Siteworx, Inc.

 Festina Lente - Gaius Julius

 ___
 Zope-Dev maillist  -  Zope-Dev@zope.org 
 http://mail.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  ** (Related lists -  
 http://mail.zope.org/mailman/listinfo/zope-announce
  http://mail.zope.org/mailman/listinfo/zope )


___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Memory Leak Question

2005-10-31 Thread Dan Pozmanter



Ahoy,

 I was 
poking around in _Acquisition.c, and noticed my memory usage 
spiked.
Returning to the 
vanilla code (2.8.2-final), I noticed the same spike (just less 
pronounced).
(If you refresh the 
page a few gazillion times this pops up).

I would not 
categorize this as an actual bug report (I'd like to test more rigorously with a 
script to verify a leak),
but after reading 
the note on Changes for 2.8.x in the to do section, I was 
curious:
Is this a known 
issue?

If so, has there 
been any headway, and can I be of any assistance in resolving 
it?

___

Daniel PozmanterSiteworx, 
Inc.703-520-1550 x209

"Festina Lente" - Gaius 
Julius

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Memory Leak Question

2005-10-31 Thread Chris McDonough
Nobody has yet reported a memory leak symptom against any particular  
piece of Zope 2.8.X yet, AFAIK.


On Oct 31, 2005, at 6:34 PM, Dan Pozmanter wrote:


Ahoy,

   I was poking around in _Acquisition.c, and noticed my memory  
usage spiked.
Returning to the vanilla code (2.8.2-final), I noticed the same  
spike (just less pronounced).

(If you refresh the page a few gazillion times this pops up).

I would not categorize this as an actual bug report (I'd like to  
test more rigorously with a script to verify a leak),
but after reading the note on Changes for 2.8.x in the to do  
section, I was curious:

Is this a known issue?

If so, has there been any headway, and can I be of any assistance  
in resolving it?


___

Daniel Pozmanter
Siteworx, Inc.
703-520-1550 x209

Festina Lente - Gaius Julius

___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )



___
Zope-Dev maillist  -  Zope-Dev@zope.org
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce

http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Memory leak, the resolution

2003-08-29 Thread Leonardo Rochael Almeida
On Thu, 2003-08-28 at 17:13, Shane Hathaway wrote:
 Leonardo Rochael Almeida wrote:
  Let's assume we're not hitting a 2.6 bug. we could be hitting a 2.6
  feature :-) I've read Casey's article about optimizing ZCatalog and I
  know there has been a lot of ZCatalog optimization work. Has there been
  any changes in 2.6 (in ZCatalog or thread cache or anywhere else) that
  was a tradeoff of memory for speed?
 
 Not that I know of.  To the contrary, I think--we sometimes can't do an 
 optimization because it would consume extra memory.

Well, as long as I can put an upper bound on the consumed memory, and
the tradeoff is reasonable, I'd be glad to trade memory for speed.

  You mean that one ZCatalog object is in the cache, and (indirectly)
  attached to it are thousands of DateTimes? Then we have a huge problem.
 
 Why?  Although the catalog manages those DateTime objects, the DateTimes 
 are in many ZODB records.

The problem is: there should be a more direct relation between the
number of allowed objects in the cache and the number of refcounts. As
you say, the DateTimes are in ZODB Records, so if I see a DateTime
refcount of, say 120k, I should be seeing at least that many objects in
the caches, but I was seeing under 3k objects, unless there's something
else that could be counted as a refcount multiplying factor, say if
something was keeping a reference to the class for every instance
besides the instance itself.

  I was counting on the objects in cache parameter to put an upper bound
  in Zope memory consumption, even if this upper bound was a little fuzzy.
 
 In the past, the cache size parameter was very fuzzy.  It was not an 
 upper bound.  Now, it is not only an upper bound, it also tends to act 
 as a lower bound.  (Note that the cache is still allowed to grow 
 indefinitely within the scope of a request, however.)


  [..] but for now we need to deal with the emergency that is the fact that
  migrating to 2.6.1 put us in the very hot spot of not having a way to
  put an upper bound to Zope memory consumption.
 
 I would characterize it the other way around; there was no way to 
 establish a hard boundary until now.  I suggest you reduce the ZODB 
 cache size to 2000 (or less) and see if the problem goes away.

The first time you sugested this I didn't even bother because we were
seeing our memory down the drain even while the caches where still at
1600 objects each (we had our limit at 5000).

But as we ran out of options, and seeing that flushing the cache got us
rid of the DateTimes. We decided to test the possibility of keeping Zope
thrashing its cache, even at the expense of performance, to see if it
could shed the DateTimes. 

So we set the cache limit to 1000 objects, and to our amazement, Zope
sat comfortably at 150MB RSS, and the performance was very good. Some
heavy ZCatalog pages still send Zope for a walk for a few seconds (for
those and all other requests) but the site becomes responsive again in
under 20 seconds. Not only that, but the DateTime refcount isn't even
among the top 5 (the BTrees are winning there now).

So we now have breathing space to do the great ZCatalog reform.

  Anyone know off the top of their head how do I get to the object caches?
 
 app._p_jar._cache will get you started.  It is dictionary-like.

Thanks, as soon as we get more RAM in the machine, I'll see if I can
switch the cache to a high number again and see where all those
DateTimes are hanging.

Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Memory Leak Problem

2002-03-14 Thread Andre Schubert

Hi all,

I have isolated the memory leak problem on a naked 2.3.3 Zope.
I have on ZPsycopgDA and two SQL Methods, one SQL Method contains
DTML-IFs and DTML-ELSEs and REQUEST.sets
the other SQL Method contains the rendered query of the first, this
means there is pure sql in it.
Both methods cache there results 60 secs.
Zope runs with 4 threads.

I wrote a test-script wich opens 4 threads and each thread call the
second SQL Methods manage_test via wget.

After a restart my zope-processes are around 12MB.
Then i start my script and before i receive any result all processes are
grown up to 26MB.

Even if after i start my script nothing happens, but before i receive
the result the processes grow up,
maybe i think this is the time where zope gets the results from the
database and
puts it into the Results-Class( i am right at this point ?).

But is the problem in the ZSQLMethod( where the results were cached ) or
the ZPsycopgDA ??

Regards,

-- 
Andre SchubertEMail:   [EMAIL PROTECTED]
  Tel: 03774 6625-78
km3 teledienst GmbH   Fax: 03774 6625-79

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-14 Thread Toby Dickenson

On Thu, 14 Mar 2002 13:50:13 +0100, Andre Schubert
[EMAIL PROTECTED] wrote:

After a restart my zope-processes are around 12MB.

Sounds about right.

Then i start my script and before i receive any result all processes are
grown up to 26MB.

Thats not unreasonable for the working set of zope. I have no reason
to suspect a leak.

Im not sure what operating system you are using. If its linux then
maybe you are misinterpreting the output of ps or top: The 26M is
listed next to each process, but its the *same* 26M in each row of the
table. In other words, your system is using 26M total for zope, not
26*5 (assuming you have 5 processes).

If Zope grows by 12M for each request then thats a different scenario.

12M growth on the *first* request is quite normal.

Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-14 Thread Andre Schubert

Toby Dickenson schrieb:
 
 On Thu, 14 Mar 2002 13:50:13 +0100, Andre Schubert
 [EMAIL PROTECTED] wrote:
 
 After a restart my zope-processes are around 12MB.
 
 Sounds about right.
 
 Then i start my script and before i receive any result all processes are
 grown up to 26MB.
 
 Thats not unreasonable for the working set of zope. I have no reason
 to suspect a leak.
 
 Im not sure what operating system you are using. If its linux then

RedHat Immunix 6.2

 maybe you are misinterpreting the output of ps or top: The 26M is
 listed next to each process, but its the *same* 26M in each row of the
 table. In other words, your system is using 26M total for zope, not
 26*5 (assuming you have 5 processes).
 
 If Zope grows by 12M for each request then thats a different scenario.
 
 12M growth on the *first* request is quite normal.

Yes, i believe, but every time i run that script the memory usage of
zope( all processes ) grows up.
If the script was ended and zope grows up about 1MB would i think that
each request for the ZSQL Methods grows up zope
about 250K and 250K * 4 = 1MB.
My problem is that every time after running the script zope grows up.
Sometimes zope grows up about 1 or 2MB and sometimes only 100K, but zope
is growing up every time a run my test-script.

On our production server the last time zope grows up to 100MB in 6
hours, and maybe thats not so good.

 
 Toby Dickenson
 [EMAIL PROTECTED]
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-14 Thread Andre Schubert

Andre Schubert schrieb:

If have changed the that it skips the ZSQLMethod and sends the query
used in the ZSQLMethod direct
to the manage_test function from the ZPsycopgDA with the result that
zope grows up
after running the script.
If it is possible that the mem-leak is somewhere in the DA, then i post
this message to the psycopg mailinglist.

Thanks

 
 Hi all,
 
 I have isolated the memory leak problem on a naked 2.3.3 Zope.
 I have on ZPsycopgDA and two SQL Methods, one SQL Method contains
 DTML-IFs and DTML-ELSEs and REQUEST.sets
 the other SQL Method contains the rendered query of the first, this
 means there is pure sql in it.
 Both methods cache there results 60 secs.
 Zope runs with 4 threads.
 
 I wrote a test-script wich opens 4 threads and each thread call the
 second SQL Methods manage_test via wget.
 
 After a restart my zope-processes are around 12MB.
 Then i start my script and before i receive any result all processes are
 grown up to 26MB.
 
 Even if after i start my script nothing happens, but before i receive
 the result the processes grow up,
 maybe i think this is the time where zope gets the results from the
 database and
 puts it into the Results-Class( i am right at this point ?).
 
 But is the problem in the ZSQLMethod( where the results were cached ) or
 the ZPsycopgDA ??
 
 Regards,
 
 --
 Andre SchubertEMail:   [EMAIL PROTECTED]
   Tel: 03774 6625-78
 km3 teledienst GmbH   Fax: 03774 6625-79
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Toby Dickenson

On Tue, 12 Mar 2002 18:38:16 +0100, Oliver Bleutgen [EMAIL PROTECTED]
wrote:

Acquisition.ImplicitAcquirerWrapper: 42442

That class is used to glue together acquisition content chains. Being
top of the list indicates that you have been leaking an acquisition
context chain. 

That doesnt mean that the bug relates to acquisition; acquisition just
means that the leak is more severe than otherwise (because it keeps
the whole context chain around).

OFS.DTMLMethod.DTMLMethod: 3989

42442/3989 = 10.6.
That indicates that you are leaking an acquisition chain of roughly 10
objects.

ZPublisher.HTTPRequest.HTTPRequest: 3941
ZServer.HTTPResponse.ZServerHTTPResponse: 3934
ZPublisher.BaseRequest.RequestContainer: 3931
AccessControl.DTML.TemplateDict: 3930

The numbers for those objects are all roughly 3900. These objects are
the ones referenced by that acquisition chain.

I recommend you look further down that list, until you find a number
that is much less than 3900. All of the objects above that mark are
pieces of the jigsaw puzzle that will tell you *what* is being leaked.
Thats where I would start meditating to work out *how* its being
leaked.

doesn't surely look too good, heh? ;-)

No way. Memory leaks are fun.

;-)



Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Oliver Bleutgen

Toby Dickenson wrote:
 On Tue, 12 Mar 2002 18:38:16 +0100, Oliver Bleutgen [EMAIL PROTECTED]
 wrote:
 
 
Acquisition.ImplicitAcquirerWrapper: 42442

 
 That class is used to glue together acquisition content chains. Being
 top of the list indicates that you have been leaking an acquisition
 context chain. 
 
 That doesnt mean that the bug relates to acquisition; acquisition just
 means that the leak is more severe than otherwise (because it keeps
 the whole context chain around).
 
 
OFS.DTMLMethod.DTMLMethod: 3989

 
 42442/3989 = 10.6.
 That indicates that you are leaking an acquisition chain of roughly 10
 objects.
 
 
ZPublisher.HTTPRequest.HTTPRequest: 3941
ZServer.HTTPResponse.ZServerHTTPResponse: 3934
ZPublisher.BaseRequest.RequestContainer: 3931
AccessControl.DTML.TemplateDict: 3930

 
 The numbers for those objects are all roughly 3900. These objects are
 the ones referenced by that acquisition chain.
 
 I recommend you look further down that list, until you find a number
 that is much less than 3900. All of the objects above that mark are
 pieces of the jigsaw puzzle that will tell you *what* is being leaked.
 Thats where I would start meditating to work out *how* its being
 leaked.

Ah, thanks for the nice explanation, Toby. I don't want to abuse your or 
others time too much, but since this might be interesting in general 
too, I'll give some more data. To be clear, I don't want to really trace 
down that leak, we live with it for nearly a year now, the server will 
be replaced soon, but I'm smelling some zen and my appetite is whet.

- I did paste only the objects which have such a high count (next one 
was approx 200).
- This server (2.3.3) has no special products on it, just 2 Folderish 
ZClasses, which are heavily used (they construct the whole site).

So, can we rule out that there's a bug in the zclasses machinery which 
is the cause for this? I.e. should there be some zclass related object 
in the above list otherwise?

Is it right that in Zope 2.3.x (and others?) usage of
dtml-var REQUEST.set(...) can lead to such leakage?

Is there a description somewhere what the basic causes of such leakages 
are? I.e. only bugs in python c-code/zope c-code?

Is there some text about what to not do in python product development to 
avoid introducing leakage? For example, on #zope someone pointed out to 
me that storing acquisition wrappers in the ZODB will lead to leakage.

cheers,
oliver












___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Toby Dickenson

On Wed, 13 Mar 2002 11:37:00 +0100, Oliver Bleutgen [EMAIL PROTECTED]
wrote:

- I did paste only the objects which have such a high count (next one 
was approx 200).

Hmm. Ill have to ponder what that means.

- This server (2.3.3) has no special products on it, just 2 Folderish 
ZClasses, which are heavily used (they construct the whole site).

So, can we rule out that there's a bug in the zclasses machinery which 
is the cause for this?

No, thats not impossible

 I.e. should there be some zclass related object 
in the above list otherwise?

Not necessarily. The class with the bug need not leave an instance of
himself tied up in his immortal cyclic trash.

Is it right that in Zope 2.3.x (and others?) usage of
dtml-var REQUEST.set(...) can lead to such leakage?

Yes, thats true in all versions.

Is there a description somewhere what the basic causes of such leakages 
are? I.e. only bugs in python c-code/zope c-code?

No, its possible for a bug in through-the-web edited dtml to cause
this.

Is there some text about what to not do in python product development to 
avoid introducing leakage? For example, on #zope someone pointed out to 
me that storing acquisition wrappers in the ZODB will lead to leakage.

In old versions (roughly before 2.3 or 2.4; I forget exactly when) it
was possible to create a cyles involving a persistent object having a
reference to an acquisition wrapper. Today that raises an exception.



Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Andre Schubert

Andre Schubert schrieb:
 
 Andy McKay schrieb:
 
  Didnt Shane do a leak finder product that can help debug this?
  http://www.zope.org/Members/hathawsh
 
 I will install the LeakFinder product on the server and look what
 happens

If have installed LeakFinder this morning and restarted zope.
My processes are grown up from 48MB to 78MB in about 4 hours, it never
was so much ( the last time 10MB per week! )
In this time i got the following top 10 refcounts from leakfinder.

OFS.Image.Image 36319
DocumentTemplate.DT_Util.Eval 9802
DateTime.DateTime.DateTime 6539
OFS.DTMLDocument.DTMLDocument 5454
OFS.Folder.Folder 5386
OFS.DTMLMethod.DTMLMethod 4850
DocumentTemplate.DT_In.InClass 1725
Products.ZSQLMethods.SQL.SQL 996
Products.TransparentFolders.TransparentFolder.TransparentFolder 458
DocumentTemplate.DT_Let.Let 398

Hope this list helps ...

Thanks

 
  On March 12, 2002 08:42 am, Chris McDonough wrote:
   Finding memory leaks is an exercise in binary search.  Isolate half of
   the Zope (and Product) code and find out if it leaks.  If it doesn't, you
   know the problem is in the other half. ;-)  Ad infinitum.  I'm sorry this
   is the case, but it almost always boils down to this particular recipe of
   diagnostics.
  --
Andy McKay
 
  ___
  Zope-Dev maillist  -  [EMAIL PROTECTED]
  http://lists.zope.org/mailman/listinfo/zope-dev
  **  No cross posts or HTML encoding!  **
  (Related lists -
   http://lists.zope.org/mailman/listinfo/zope-announce
   http://lists.zope.org/mailman/listinfo/zope )
 
 --
 Andre SchubertEMail:   [EMAIL PROTECTED]
   Tel: 03774 6625-78
 km3 teledienst GmbH   Fax: 03774 6625-79
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

-- 
Andre SchubertEMail:   [EMAIL PROTECTED]
  Tel: 03774 6625-78
km3 teledienst GmbH   Fax: 03774 6625-79

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Oliver Bleutgen

One more question then I'll shut up ;-).

Toby Dickenson wrote:

Is there a description somewhere what the basic causes of such leakages 
are? I.e. only bugs in python c-code/zope c-code?

 
 No, its possible for a bug in through-the-web edited dtml to cause
 this.


Waah, this is the first time I hear that. That means that even if 
zope/python would be bug-free, it's possible to write dtml that causes a 
leak?
Is there a (non-esoteric) example for that?


cheers,
oliver


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-13 Thread Toby Dickenson

On Wed, 13 Mar 2002 13:46:27 +0100, Oliver Bleutgen [EMAIL PROTECTED]
wrote:

Waah, this is the first time I hear that. That means that even if 
zope/python would be bug-free, it's possible to write dtml that causes a 
leak?
Is there a (non-esoteric) example for that?

(untested, but the principal is sound)

dtml-call REQUEST.set('foo',x)

where x is some object whose acquisition context includes the REQUEST
object



Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Memory Leak Problem

2002-03-12 Thread Andre Schubert

Hi all,

i have a little problem with my production server.
The memory usage of the zope processes running on this server are
growing up
100K a day upto 1MB a day.
How can i track down the problem.

Zope is 2.3.3
Python is 1.5.2
OS is RedHat Immunix 6.2

Hope somebody could help... 

Thanks,

-- 
Andre SchubertEMail:   [EMAIL PROTECTED]
  Tel: 03774 6625-78
km3 teledienst GmbH   Fax: 03774 6625-79

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-12 Thread Chris McDonough

Finding memory leaks is an exercise in binary search.  Isolate half of the
Zope (and Product) code and find out if it leaks.  If it doesn't, you know
the problem is in the other half. ;-)  Ad infinitum.  I'm sorry this is the
case, but it almost always boils down to this particular recipe of
diagnostics.

- Original Message -
From: Andre Schubert [EMAIL PROTECTED]
To: zope [EMAIL PROTECTED]
Sent: Tuesday, March 12, 2002 10:57 AM
Subject: [Zope-dev] Memory Leak Problem


 Hi all,

 i have a little problem with my production server.
 The memory usage of the zope processes running on this server are
 growing up
 100K a day upto 1MB a day.
 How can i track down the problem.

 Zope is 2.3.3
 Python is 1.5.2
 OS is RedHat Immunix 6.2

 Hope somebody could help...

 Thanks,

 --
 Andre SchubertEMail:   [EMAIL PROTECTED]
   Tel: 03774 6625-78
 km3 teledienst GmbH   Fax: 03774 6625-79

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-12 Thread Oliver Bleutgen

 Hi all,
 
 i have a little problem with my production server.
 The memory usage of the zope processes running on this server are
 growing up
 100K a day upto 1MB a day.
 How can i track down the problem.
[snip]


Chris McDonough wrote:
 Finding memory leaks is an exercise in binary search.  Isolate half of the
 Zope (and Product) code and find out if it leaks.  If it doesn't, you know
 the problem is in the other half. ;-)  Ad infinitum.  I'm sorry this is the
 case, but it almost always boils down to this particular recipe of
 diagnostics.

A question and one remark.
Wouldn't staring at the refcounts in the debug panel also give some 
info? At least for selfmade products?

Something like:

Acquisition.ImplicitAcquirerWrapper: 42442
OFS.DTMLMethod.DTMLMethod: 3989
ZPublisher.HTTPRequest.HTTPRequest: 3941
ZServer.HTTPResponse.ZServerHTTPResponse: 3934
ZPublisher.BaseRequest.RequestContainer: 3931
AccessControl.DTML.TemplateDict: 3930

doesn't surely look too good, heh? ;-)
Yes this is from a production server, which now get's restarted as it 
occupies 130MB (up from 15MB in 10 days).


The remark, Andre, Chris has written a very nice product, called 
AutoLance which helps by monitoring zope's memory usage and restarting 
if a configurable threshold is reached.

http://www.zope.org/Members/mcdonc/Products/AutoLance

We use it on a production server.


cheers,
oliver







___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-12 Thread Chris McDonough

 A question and one remark.
 Wouldn't staring at the refcounts in the debug panel also give some
 info? At least for selfmade products?

Maybe... depends on your level of Zope Zen.  It's hard to tell which classes
should and should not have high refcounts.  For example, in your sample
output, I know that high numbers of references to ImplicitAcquirerWrappers,
HTTPRequests, ZServerHTTPResponse, and RequestContainer objects likely
indicate a leak of REQUEST objects somewhere in your code.  OTOH, the high
number of references to DTMLMethods might be totally normal.  But I only
know these things because I know Zope maybe better than most folks do, and I
just...  well... *know*.  ;-)  It's hard to explain the reasons I know these
things to other folks (although I started on something like it at
http://www.zope.org/Members/mcdonc/HowTos/PreventAndDebugMemLeaks
(unfinished).

That said, even if I took a day to finish my unfinished HowTo, the only bit
of wisdom that I could consistently espouse given Control_Panel - Debug
output is that indeed you do have a leak.  So you'd know you have a leak
(which you knew in the first place).  How do you fix it?  I don't know. ;-)
That's where the binary search comes in.  Shane's LeakFinder can make the
binary search a bit easier by recording states between leakfinding runs, but
it's still mainly an exercise in binary search.

 The remark, Andre, Chris has written a very nice product, called
 AutoLance which helps by monitoring zope's memory usage and restarting
 if a configurable threshold is reached.

If you can't solve the problem at least treat the symptoms. ;-)

- C



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-12 Thread Andre Schubert

Andy McKay schrieb:
 
 Didnt Shane do a leak finder product that can help debug this?
 http://www.zope.org/Members/hathawsh

I will install the LeakFinder product on the server and look what
happens

 
 On March 12, 2002 08:42 am, Chris McDonough wrote:
  Finding memory leaks is an exercise in binary search.  Isolate half of
  the Zope (and Product) code and find out if it leaks.  If it doesn't, you
  know the problem is in the other half. ;-)  Ad infinitum.  I'm sorry this
  is the case, but it almost always boils down to this particular recipe of
  diagnostics.
 --
   Andy McKay
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

-- 
Andre SchubertEMail:   [EMAIL PROTECTED]
  Tel: 03774 6625-78
km3 teledienst GmbH   Fax: 03774 6625-79

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory Leak Problem

2002-03-12 Thread Andre Schubert

Oliver Bleutgen schrieb:
 
  Hi all,
  
  i have a little problem with my production server.
  The memory usage of the zope processes running on this server are
  growing up
  100K a day upto 1MB a day.
  How can i track down the problem.
 [snip]
 
 Chris McDonough wrote:
  Finding memory leaks is an exercise in binary search.  Isolate half of the
  Zope (and Product) code and find out if it leaks.  If it doesn't, you know
  the problem is in the other half. ;-)  Ad infinitum.  I'm sorry this is the
  case, but it almost always boils down to this particular recipe of
  diagnostics.
 
 A question and one remark.
 Wouldn't staring at the refcounts in the debug panel also give some
 info? At least for selfmade products?
 
 Something like:
 
 Acquisition.ImplicitAcquirerWrapper: 42442
 OFS.DTMLMethod.DTMLMethod: 3989
 ZPublisher.HTTPRequest.HTTPRequest: 3941
 ZServer.HTTPResponse.ZServerHTTPResponse: 3934
 ZPublisher.BaseRequest.RequestContainer: 3931
 AccessControl.DTML.TemplateDict: 3930

The Toprefcounts are:
Acquisition.ImplicitAcquirerWrapper: 2981
DocumentTemplate.DT_Util.Eval: 1379
OFS.Image.Image: 1194
ZPublisher.HTTPRequest.HTTPRequest: 916
ZServer.PCGIServer.PCGIResponse: 906
ZPublisher.BaseRequest.RequestContainer: 906
ExtensionClass: 361
DocumentTemplate.DT_In.InClass: 263
OFS.Folder.Folder: 250
OFS.DTMLDocument.DTMLDocument: 238
DocumentTemplate.DT_Var.Var: 207
OFS.DTMLMethod.DTMLMethod: 196
AccessControl.PermissionRole.PermissionRole: 184
App.special_dtml.DTMLFile: 182
Products.ZGadflyDA.gadfly.kjParser.ParseRule: 161
re.RegexObject: 153
Shared.DC.Scripts.Signature.FuncCode: 143
and so on.

My selfmade products are near the bottom of the list...

Thanks

 
 doesn't surely look too good, heh? ;-)
 Yes this is from a production server, which now get's restarted as it
 occupies 130MB (up from 15MB in 10 days).
 
 The remark, Andre, Chris has written a very nice product, called
 AutoLance which helps by monitoring zope's memory usage and restarting
 if a configurable threshold is reached.
 
 http://www.zope.org/Members/mcdonc/Products/AutoLance
 
 We use it on a production server.
 
 cheers,
 oliver
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

-- 
Andre SchubertEMail:   [EMAIL PROTECTED]
  Tel: 03774 6625-78
km3 teledienst GmbH   Fax: 03774 6625-79

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] Memory leak (2)

2001-04-25 Thread Andy McKay

Just occured to me if Im leaking REQUEST it would be nice to see exactly
which requests im leaking. But is that really covered by the -M logger, if a
request fails to be closed in there does that really translate to a leaked
request? I had a look in the code and it wasn't obvious to me

Cheers.
--
  Andy McKay.




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Memory leak (2)

2001-04-25 Thread Andy McKay


 It's not covered by the data output into the -M log.  Objects can leak
 without requests hanging or taking a long time.  But you might be able to
 turn on profiling (set the PROFILE_PUBLISHER env var to a filename and
 restart Zope) on a *test* box (do not do it in production, it drastically
 effects performance), and look at the profile output.  If the memory loss
is
 severe, it might help to investigate the methods that have the largest
 ncalls value (the most frequently called methods).  That said, there's
no
 globally easy way to diagnose a memory leak.  It's a matter of exclusion.

Well there goes that idea. Forgot about profiling, thanks.
--
  Andy McKay.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] memory leak

2001-04-13 Thread Karl Anderson

Marco Nova [EMAIL PROTECTED] writes:

 I've replaced ParsedXML's ExpatBuilder with the pyXML package and I used the
 sax parser without modifing the code (except for the import), this is the
 refcounts results.
 
 Class  April 12, 2001 11:55 am  April 12, 2001 12:00
 
 xml.dom.NodeList.NodeList   2103  3678  +1575  
 xml.dom.Text.Text   1263  2208  +945  
 xml.dom.NamedNodeMap.NamedNodeMap507   885  +378  
 xml.dom.Element.Element  499   871  +372  
 xml.dom.Attr.Attr339   591  +252  
 xml.dom.DocumentType.DocumentType  710  +3  
 xml.dom.Document.Document  710  +3
 
 xml.dom.Document.Document increments by 1 each time an xml is procesed (I
 tried to add a del doc at the end but it's ineffective).
 
 So the problem is not ParsedXML but Zope itself or my bad methods.

Not necessarily - it's been mentioned that PyXML requires releasing
nodes after they're done, so I'd expect this if you replaced
ParsedXML's DOM with PyXML's without adding the release.

-- 
Karl Anderson  [EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



RE: [Zope-dev] memory leak

2001-04-12 Thread Marco Nova

 In either case, it seems like you shouldn't be using ParsedXML's
 ExpatBuilder, which is an interface to PyExpat which is made to create
 ParsedXML's DOM objects.  You should be using your own PyExpat
 interface to parse this information from your XML input, without the
 overhead of generating a DOM tree that you're just going to throw
 away.  The interface is documented in the XML-SIG topic area on
 python.org, and our ExpatBuilder should be a good example on how to
 interface with our PyExpat installation - just don't build the DOM if
 you don't use it.
 
 Regardless of what you *should* be doing :), or if I'm misreading you,
 if you aren't keeping references to the DOM objects being created,
 then they should be reclaimed by the GC, and you do seem to have DOM
 objects lying around.  I'd like to see your code, is it short enough
 to post?  Or you can chuck it into the collector or ParsedXML tracker.
 
 For debugging purporses, I'd suggest simplifying by supplying an XML
 string directly, rather than from an external server.

I've replaced ParsedXML's ExpatBuilder with the pyXML package and I used the
sax parser without modifing the code (except for the import), this is the
refcounts results.

Class  April 12, 2001 11:55 am  April 12, 2001 12:00

xml.dom.NodeList.NodeList   2103  3678  +1575  
xml.dom.Text.Text   1263  2208  +945  
xml.dom.NamedNodeMap.NamedNodeMap507   885  +378  
xml.dom.Element.Element  499   871  +372  
xml.dom.Attr.Attr339   591  +252  
xml.dom.DocumentType.DocumentType  710  +3  
xml.dom.Document.Document  710  +3

xml.dom.Document.Document increments by 1 each time an xml is procesed (I
tried to add a del doc at the end but it's ineffective).

So the problem is not ParsedXML but Zope itself or my bad methods.

Regards,

- mn

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



RE: [Zope-dev] memory leak

2001-04-11 Thread Marco Nova

 have you checked the debug page in the control panel and 
 noted any refcounts
 increasing?

Class  April 10, 2001 9:17 am  April 11, 2001 8:34 am

Products.ParsedXML.DOM.Core.Text  3241  27318  +24077  
Products.ParsedXML.DOM.Core.Element   1777  14493  +12716  
Products.ParsedXML.DOM.ExpatBuilder.ExpatBuilder  81  1506  +1425  
Products.ParsedXML.Printer.PrintVisitor   3  107  +104  
Products.ParsedXML.DOM.Core.Document  10  112  +102  

Yes, I've a lot of refcounts leaks.

 any chance that this is the same problem that andy is running 
 into (read his
 posts above)?

I dunno because of my very lack of knowledge of Zope.
All of these refcounts leaks means that the Product is doing something wrong
or that I'm using in the wrong manner ?

- mn

 
 -d
 
 On Tue, Apr 10, 2001 at 05:27:28PM +0200, Marco Nova wrote:
  Hello guys,
  
  It seems that I've some memory leak problems and I want to 
 understand where
  I've to look for.
  
  Scenario:
  
  I've an IIS web server that generates an XML page (taking data from
  SqlServer);
  The Zope web server must retrieve the XML page, manage the 
 data and render
  them in HTML (I cant only apply a XSL stylesheet to XML).
  
  The Facts:
  
  I've written a external method that - using httplib - reads 
 the XML page and
  return a string that is the raw XML sources, than I've 
 written another
  external method that use ExpatBuilder of 
 Products.ParsedXML.DOM to parse the
  XML an generate a list of [(id - {mapping})] which will be 
 accessed with a
  dtml-in tag, so in the sequence-key I've access to the ID and with
  sequence-item I can access to the mapping object.
  I'm using latest version of ZOPE (2.3.1, the binary one 
 with Python 1.5.2)
  and ParsedXML.
  
  The Problem:
  
  It seems that every time the method retrieve the page or 
 parse the XML I
  loose a bunch of kilobytes. Because the methods are very 
 simple (100 lines
  of code in total) and because I didn't create any class for 
 handling this
  workflow I can't understand where I can loose bytes.
  
  The Questions:
  
  Is the list generated for the dtml-in tag freed 
 automatically by ZOPE ?
  Is there a Garbage Collector somewhere that must be activated ?
  Should I avoid the use of the ExpatBuilder for XML processing ?
  
  Thanks for listening my newbies questions.
  
  
  Best Regards,
  
  - mn
  
  ___
  Zope-Dev maillist  -  [EMAIL PROTECTED]
  http://lists.zope.org/mailman/listinfo/zope-dev
  **  No cross posts or HTML encoding!  **
  (Related lists - 
   http://lists.zope.org/mailman/listinfo/zope-announce
   http://lists.zope.org/mailman/listinfo/zope )
 
 -- 
 Dyon Balding [EMAIL PROTECTED]
 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] memory leak

2001-04-11 Thread Chris McDonough

Hi Marco,

This looks like a potential bug in either ParsedXML or the code you're using
to manipulate the components within ParsedXML.

We'd like to find out!  ;-)  Do you think you can file a collector report
with enough information in it to reproduce this behavior?

Many thanks,

- C

- Original Message -
From: "Marco Nova" [EMAIL PROTECTED]
To: "Zope-Dev (E-mail)" [EMAIL PROTECTED]
Sent: Wednesday, April 11, 2001 2:29 AM
Subject: RE: [Zope-dev] memory leak


  have you checked the debug page in the control panel and
  noted any refcounts
  increasing?

 Class  April 10, 2001 9:17 am  April 11, 2001 8:34 am

 Products.ParsedXML.DOM.Core.Text  3241  27318  +24077
 Products.ParsedXML.DOM.Core.Element   1777  14493  +12716
 Products.ParsedXML.DOM.ExpatBuilder.ExpatBuilder  81  1506  +1425
 Products.ParsedXML.Printer.PrintVisitor   3  107  +104
 Products.ParsedXML.DOM.Core.Document  10  112  +102

 Yes, I've a lot of refcounts leaks.

  any chance that this is the same problem that andy is running
  into (read his
  posts above)?

 I dunno because of my very lack of knowledge of Zope.
 All of these refcounts leaks means that the Product is doing something
wrong
 or that I'm using in the wrong manner ?

 - mn

 
  -d
 
  On Tue, Apr 10, 2001 at 05:27:28PM +0200, Marco Nova wrote:
   Hello guys,
  
   It seems that I've some memory leak problems and I want to
  understand where
   I've to look for.
  
   Scenario:
  
   I've an IIS web server that generates an XML page (taking data from
   SqlServer);
   The Zope web server must retrieve the XML page, manage the
  data and render
   them in HTML (I cant only apply a XSL stylesheet to XML).
  
   The Facts:
  
   I've written a external method that - using httplib - reads
  the XML page and
   return a string that is the raw XML sources, than I've
  written another
   external method that use ExpatBuilder of
  Products.ParsedXML.DOM to parse the
   XML an generate a list of [(id - {mapping})] which will be
  accessed with a
   dtml-in tag, so in the sequence-key I've access to the ID and with
   sequence-item I can access to the mapping object.
   I'm using latest version of ZOPE (2.3.1, the binary one
  with Python 1.5.2)
   and ParsedXML.
  
   The Problem:
  
   It seems that every time the method retrieve the page or
  parse the XML I
   loose a bunch of kilobytes. Because the methods are very
  simple (100 lines
   of code in total) and because I didn't create any class for
  handling this
   workflow I can't understand where I can loose bytes.
  
   The Questions:
  
   Is the list generated for the dtml-in tag freed
  automatically by ZOPE ?
   Is there a Garbage Collector somewhere that must be activated ?
   Should I avoid the use of the ExpatBuilder for XML processing ?
  
   Thanks for listening my newbies questions.
  
  
   Best Regards,
  
   - mn
  
   ___
   Zope-Dev maillist  -  [EMAIL PROTECTED]
   http://lists.zope.org/mailman/listinfo/zope-dev
   **  No cross posts or HTML encoding!  **
   (Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
 
  --
  Dyon Balding [EMAIL PROTECTED]
 

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



RE: [Zope-dev] memory leak

2001-04-11 Thread Marco Nova

 -Original Message-
 From: Chris McDonough [mailto:[EMAIL PROTECTED]]
 Sent: mercoled 11 aprile 2001 9.04
 To: Marco Nova; Zope-Dev (E-mail)
 Subject: Re: [Zope-dev] memory leak

 This looks like a potential bug in either ParsedXML or the 
 code you're using
 to manipulate the components within ParsedXML.
 
 We'd like to find out!  ;-)  Do you think you can file a 
 collector report
 with enough information in it to reproduce this behavior?

I can send you the code for the two external method I use, they only need a
webserver (that could be zope too) that serve an XML page (I can provide the
XML the methods needs) and the bunch of dtml methods I use to show the XML.
Anyway I will try to understand what's happening with ParsedXML and my code.

- mn

 
 - Original Message -
 From: "Marco Nova" [EMAIL PROTECTED]
 To: "Zope-Dev (E-mail)" [EMAIL PROTECTED]
 Sent: Wednesday, April 11, 2001 2:29 AM
 Subject: RE: [Zope-dev] memory leak
 
 
   have you checked the debug page in the control panel and
   noted any refcounts
   increasing?
 
  Class  April 10, 2001 9:17 am  April 11, 2001 8:34 am
 
  Products.ParsedXML.DOM.Core.Text  3241  
 27318  +24077
  Products.ParsedXML.DOM.Core.Element   1777  
 14493  +12716
  Products.ParsedXML.DOM.ExpatBuilder.ExpatBuilder  81  1506  +1425
  Products.ParsedXML.Printer.PrintVisitor   3  107  +104
  Products.ParsedXML.DOM.Core.Document  10  112  +102
 
  Yes, I've a lot of refcounts leaks.
 
   any chance that this is the same problem that andy is running
   into (read his
   posts above)?
 
  I dunno because of my very lack of knowledge of Zope.
  All of these refcounts leaks means that the Product is 
 doing something
 wrong
  or that I'm using in the wrong manner ?
 
  - mn
 
  
   -d
  
   On Tue, Apr 10, 2001 at 05:27:28PM +0200, Marco Nova wrote:
Hello guys,
   
It seems that I've some memory leak problems and I want to
   understand where
I've to look for.
   
Scenario:
   
I've an IIS web server that generates an XML page 
 (taking data from
SqlServer);
The Zope web server must retrieve the XML page, manage the
   data and render
them in HTML (I cant only apply a XSL stylesheet to XML).
   
The Facts:
   
I've written a external method that - using httplib - reads
   the XML page and
return a string that is the raw XML sources, than I've
   written another
external method that use ExpatBuilder of
   Products.ParsedXML.DOM to parse the
XML an generate a list of [(id - {mapping})] which will be
   accessed with a
dtml-in tag, so in the sequence-key I've access to the 
 ID and with
sequence-item I can access to the mapping object.
I'm using latest version of ZOPE (2.3.1, the binary one
   with Python 1.5.2)
and ParsedXML.
   
The Problem:
   
It seems that every time the method retrieve the page or
   parse the XML I
loose a bunch of kilobytes. Because the methods are very
   simple (100 lines
of code in total) and because I didn't create any class for
   handling this
workflow I can't understand where I can loose bytes.
   
The Questions:
   
Is the list generated for the dtml-in tag freed
   automatically by ZOPE ?
Is there a Garbage Collector somewhere that must be activated ?
Should I avoid the use of the ExpatBuilder for XML processing ?
   
Thanks for listening my newbies questions.
   
   
Best Regards,
   
- mn
   
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )
  
   --
   Dyon Balding [EMAIL PROTECTED]
  
 
  ___
  Zope-Dev maillist  -  [EMAIL PROTECTED]
  http://lists.zope.org/mailman/listinfo/zope-dev
  **  No cross posts or HTML encoding!  **
  (Related lists -
   http://lists.zope.org/mailman/listinfo/zope-announce
   http://lists.zope.org/mailman/listinfo/zope )
 
 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] memory leak

2001-04-11 Thread Chris McDonough

The bugreport won't do us as much good if we need to set up an elaborate
testing environment... it'd be preferable if you could break it down enough
to make it happen in a single external method or pythonscript that doesn't
rely on external webservers or databases, or what-have-you... but we'll take
whatever you've got!

- C

- Original Message -
From: "Marco Nova" [EMAIL PROTECTED]
To: "'Chris McDonough'" [EMAIL PROTECTED]
Cc: "Zope-Dev (E-mail)" [EMAIL PROTECTED]
Sent: Wednesday, April 11, 2001 6:34 AM
Subject: RE: [Zope-dev] memory leak


 -Original Message-
 From: Chris McDonough [mailto:[EMAIL PROTECTED]]
 Sent: mercoled 11 aprile 2001 9.04
 To: Marco Nova; Zope-Dev (E-mail)
 Subject: Re: [Zope-dev] memory leak

 This looks like a potential bug in either ParsedXML or the
 code you're using
 to manipulate the components within ParsedXML.

 We'd like to find out!  ;-)  Do you think you can file a
 collector report
 with enough information in it to reproduce this behavior?

I can send you the code for the two external method I use, they only need a
webserver (that could be zope too) that serve an XML page (I can provide the
XML the methods needs) and the bunch of dtml methods I use to show the XML.
Anyway I will try to understand what's happening with ParsedXML and my code.

- mn


 - Original Message -
 From: "Marco Nova" [EMAIL PROTECTED]
 To: "Zope-Dev (E-mail)" [EMAIL PROTECTED]
 Sent: Wednesday, April 11, 2001 2:29 AM
 Subject: RE: [Zope-dev] memory leak


   have you checked the debug page in the control panel and
   noted any refcounts
   increasing?
 
  Class  April 10, 2001 9:17 am  April 11, 2001 8:34 am
 
  Products.ParsedXML.DOM.Core.Text  3241
 27318  +24077
  Products.ParsedXML.DOM.Core.Element   1777
 14493  +12716
  Products.ParsedXML.DOM.ExpatBuilder.ExpatBuilder  81  1506  +1425
  Products.ParsedXML.Printer.PrintVisitor   3  107  +104
  Products.ParsedXML.DOM.Core.Document  10  112  +102
 
  Yes, I've a lot of refcounts leaks.
 
   any chance that this is the same problem that andy is running
   into (read his
   posts above)?
 
  I dunno because of my very lack of knowledge of Zope.
  All of these refcounts leaks means that the Product is
 doing something
 wrong
  or that I'm using in the wrong manner ?
 
  - mn
 
  
   -d
  
   On Tue, Apr 10, 2001 at 05:27:28PM +0200, Marco Nova wrote:
Hello guys,
   
It seems that I've some memory leak problems and I want to
   understand where
I've to look for.
   
Scenario:
   
I've an IIS web server that generates an XML page
 (taking data from
SqlServer);
The Zope web server must retrieve the XML page, manage the
   data and render
them in HTML (I cant only apply a XSL stylesheet to XML).
   
The Facts:
   
I've written a external method that - using httplib - reads
   the XML page and
return a string that is the raw XML sources, than I've
   written another
external method that use ExpatBuilder of
   Products.ParsedXML.DOM to parse the
XML an generate a list of [(id - {mapping})] which will be
   accessed with a
dtml-in tag, so in the sequence-key I've access to the
 ID and with
sequence-item I can access to the mapping object.
I'm using latest version of ZOPE (2.3.1, the binary one
   with Python 1.5.2)
and ParsedXML.
   
The Problem:
   
It seems that every time the method retrieve the page or
   parse the XML I
loose a bunch of kilobytes. Because the methods are very
   simple (100 lines
of code in total) and because I didn't create any class for
   handling this
workflow I can't understand where I can loose bytes.
   
The Questions:
   
Is the list generated for the dtml-in tag freed
   automatically by ZOPE ?
Is there a Garbage Collector somewhere that must be activated ?
Should I avoid the use of the ExpatBuilder for XML processing ?
   
Thanks for listening my newbies questions.
   
   
Best Regards,
   
- mn
   
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )
  
   --
   Dyon Balding [EMAIL PROTECTED]
  
 
  ___
  Zope-Dev maillist  -  [EMAIL PROTECTED]
  http://lists.zope.org/mailman/listinfo/zope-dev
  **  No cross posts or HTML encoding!  **
  (Related lists -
   http://lists.zope.org/mailman/listinfo/zope-announce
   http://lists.zope.org/mailman/listinfo/zope )
 




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://l

RE: [Zope-dev] memory leak

2001-04-11 Thread Dieter Maurer

Marco Nova writes:
   have you checked the debug page in the control panel and 
   noted any refcounts
   increasing?
  
  Class  April 10, 2001 9:17 am  April 11, 2001 8:34 am
  
  Products.ParsedXML.DOM.Core.Text  3241  27318  +24077  
  Products.ParsedXML.DOM.Core.Element   1777  14493  +12716  
  Products.ParsedXML.DOM.ExpatBuilder.ExpatBuilder  81  1506  +1425  
  Products.ParsedXML.Printer.PrintVisitor   3  107  +104  
  Products.ParsedXML.DOM.Core.Document  10  112  +102  
  
  Yes, I've a lot of refcounts leaks.
I do not know "ParsedXML"'s DOM implementation.

However, cycles are quite common for DOM's due to the
cyclic relationship between parent and children.
Unless great care is taken, DOM tree can be highly cyclic
and would therefore not be garbage collectable by Pythons
1.5.2 reference counting garbage collector.


Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



[Zope-dev] memory leak

2001-04-10 Thread Marco Nova

Hello guys,

It seems that I've some memory leak problems and I want to understand where
I've to look for.

Scenario:

I've an IIS web server that generates an XML page (taking data from
SqlServer);
The Zope web server must retrieve the XML page, manage the data and render
them in HTML (I cant only apply a XSL stylesheet to XML).

The Facts:

I've written a external method that - using httplib - reads the XML page and
return a string that is the raw XML sources, than I've written another
external method that use ExpatBuilder of Products.ParsedXML.DOM to parse the
XML an generate a list of [(id - {mapping})] which will be accessed with a
dtml-in tag, so in the sequence-key I've access to the ID and with
sequence-item I can access to the mapping object.
I'm using latest version of ZOPE (2.3.1, the binary one with Python 1.5.2)
and ParsedXML.

The Problem:

It seems that every time the method retrieve the page or parse the XML I
loose a bunch of kilobytes. Because the methods are very simple (100 lines
of code in total) and because I didn't create any class for handling this
workflow I can't understand where I can loose bytes.

The Questions:

Is the list generated for the dtml-in tag freed automatically by ZOPE ?
Is there a Garbage Collector somewhere that must be activated ?
Should I avoid the use of the ExpatBuilder for XML processing ?

Thanks for listening my newbies questions.


Best Regards,

- mn

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] memory leak

2001-04-10 Thread Dyon Balding

have you checked the debug page in the control panel and noted any refcounts
increasing?

any chance that this is the same problem that andy is running into (read his
posts above)?

-d

On Tue, Apr 10, 2001 at 05:27:28PM +0200, Marco Nova wrote:
 Hello guys,
 
 It seems that I've some memory leak problems and I want to understand where
 I've to look for.
 
 Scenario:
 
 I've an IIS web server that generates an XML page (taking data from
 SqlServer);
 The Zope web server must retrieve the XML page, manage the data and render
 them in HTML (I cant only apply a XSL stylesheet to XML).
 
 The Facts:
 
 I've written a external method that - using httplib - reads the XML page and
 return a string that is the raw XML sources, than I've written another
 external method that use ExpatBuilder of Products.ParsedXML.DOM to parse the
 XML an generate a list of [(id - {mapping})] which will be accessed with a
 dtml-in tag, so in the sequence-key I've access to the ID and with
 sequence-item I can access to the mapping object.
 I'm using latest version of ZOPE (2.3.1, the binary one with Python 1.5.2)
 and ParsedXML.
 
 The Problem:
 
 It seems that every time the method retrieve the page or parse the XML I
 loose a bunch of kilobytes. Because the methods are very simple (100 lines
 of code in total) and because I didn't create any class for handling this
 workflow I can't understand where I can loose bytes.
 
 The Questions:
 
 Is the list generated for the dtml-in tag freed automatically by ZOPE ?
 Is there a Garbage Collector somewhere that must be activated ?
 Should I avoid the use of the ExpatBuilder for XML processing ?
 
 Thanks for listening my newbies questions.
 
 
 Best Regards,
 
 - mn
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - 
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )

-- 
Dyon Balding [EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )