Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-21 Thread Dieter Maurer
J. David Ibanez wrote at 2004-5-21 16:50 +0200:
>
>No, I don't use TranslationService. A quick glance at the code shows
>that TS uses several caches (dictionaries) which are stored in the request
>object: "_localizer_placeful_mc_cache", "_translation_service_cache"
>and "_ts_domain_cache".
>
>The values of these dictionaries are acquisition wrappers (objects returned
>by "unrestrictedTraverse" or explicitly built). Maybe the leak is there.

When these dictionaries are held in "other" (or "form" or "Lazy"),
then there is no problem -- as "other" is explicitly clear
when the request is closed.

However, when they are (direct) attributes of the request
object, then they will form cycles which almost surely
cause leakage.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-21 Thread Dieter Maurer
[EMAIL PROTECTED] wrote at 2004-5-21 09:34 -0400:
> ...
>If I remove the TranslationService instance, the problem goes away. Do you
>use this? Does anybody know of alternatives to the
>TranslationService+Localizer
>combo to enable i18n in ZPT's on Zope 2?

I am no internationalization expert. But many posts suggest
to use "PTS" (Placeless Translation Service) nowadays...

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-21 Thread J. David Ibanez
No, I don't use TranslationService. A quick glance at the code shows
that TS uses several caches (dictionaries) which are stored in the request
object: "_localizer_placeful_mc_cache", "_translation_service_cache"
and "_ts_domain_cache".
The values of these dictionaries are acquisition wrappers (objects returned
by "unrestrictedTraverse" or explicitly built). Maybe the leak is there.
[EMAIL PROTECTED] wrote:
Juan David,
Indeed, thanks for the input, I've ruled out new_publish as the cause as
well ...
Right now, I'm at the stage where I severely suspect it's got something to
do
with i18n support or translation somehow, but don't know where exactly yet.
If I remove the message_catalog, the problem goes away.
If I remove the TranslationService instance, the problem goes away. Do you
use this? Does anybody know of alternatives to the
TranslationService+Localizer
combo to enable i18n in ZPT's on Zope 2?
If I keep the above two, but delete my custom Localizer instances, the
problem persists.
I was trying to find an alternative message_catalog to use, but couldn't
find one (CMFLocalizer is based on yours).
So I'm going to try and narrow the field some more.  I *quickly* looked
the message_catalog and TranslationService code, and nothing jumped out.
I have to also look at the ZPT engine itself, maybe it's i18n
support is the cause of the problem.
I'll run further tests ...
Thanks,
J.F.
-Original Message-
From: "J. David Ibáñez" [mailto:[EMAIL PROTECTED]
Sent: May 21, 2004 8:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors

The problem I have with one of my sites seems to have a different cause, 
as it
does not show up when errors occur, but when requests are succesful.

So far I haven't been able to reproduce the problem.
You can try to modify "new_publish" to catch exceptions:
def new_publish(request, module_name, after_list, debug=0):
   id = get_ident()
   Publish._requests[id] = request
   try:
   x = Publish.old_publish(request, module_name, after_list, debug)
   except:
   del Publish._requests[id]
   raise
   else:
   try:
   del Publish._requests[id]
   except KeyError:
...
Though it is unlikely the leak to be there.
 

--
J. David Ibáñez
Founder and CTO of Itaapy <http://www.itaapy.com>
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-21 Thread J. David Ibáñez
The problem I have with one of my sites seems to have a different cause, 
as it
does not show up when errors occur, but when requests are succesful.

So far I haven't been able to reproduce the problem.
You can try to modify "new_publish" to catch exceptions:
def new_publish(request, module_name, after_list, debug=0):
   id = get_ident()
   Publish._requests[id] = request
   try:
   x = Publish.old_publish(request, module_name, after_list, debug)
   except:
   del Publish._requests[id]
   raise
   else:
   try:
   del Publish._requests[id]
   except KeyError:
...
Though it is unlikely the leak to be there.

[EMAIL PROTECTED] wrote:
Indeed, I was hoping the ref in the dictionary there was getting in the way
when old_publish was raising an exception, but I tried using a
weakref.proxy()
instead and that didn't help.
I managed to test without Localizer (It dawned on me that since I'm looking
at errors, it really doesn't matter if I don't have Localizer installed
since
not having it will generate errors, which is what I want!) ... And the
problem
diappeared ...
So it's definitely Localizer, but it's not the Globals() request patch ...
Or
at least not the act of patching it itself, that causes the problem.
*sigh*
J.F.
-Original Message-
From: Dieter Maurer [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 2:58 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors
[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
 

...
def new_publish(request, module_name, after_list, debug=0): 
  id = get_ident() 
  print "Localizer got thread id: " + str(id)
  Publish._requests[id] = request 
  print "Request dict is now: " + str(Publish._requests)   
  x = Publish.old_publish(request, module_name, after_list, debug) 
  try:
  del Publish._requests[id]
  except KeyError:
...
   

This code cannot leak requests in large numbers (though it may
leak a few requests for some time).
Reason: Zope does not normally create new threads. This implies
that thread_ids are reused and thereby old requests flushed from
the dict.
 


--
J. David Ibáñez
Founder and CTO of Itaapy <http://www.itaapy.com>
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-21 Thread Jean-Francois . Doyon
Juan David,

Indeed, thanks for the input, I've ruled out new_publish as the cause as
well ...

Right now, I'm at the stage where I severely suspect it's got something to
do
with i18n support or translation somehow, but don't know where exactly yet.

If I remove the message_catalog, the problem goes away.

If I remove the TranslationService instance, the problem goes away. Do you
use this? Does anybody know of alternatives to the
TranslationService+Localizer
combo to enable i18n in ZPT's on Zope 2?

If I keep the above two, but delete my custom Localizer instances, the
problem persists.

I was trying to find an alternative message_catalog to use, but couldn't
find one (CMFLocalizer is based on yours).

So I'm going to try and narrow the field some more.  I *quickly* looked
the message_catalog and TranslationService code, and nothing jumped out.
I have to also look at the ZPT engine itself, maybe it's i18n
support is the cause of the problem.

I'll run further tests ...

Thanks,
J.F.

-Original Message-
From: "J. David Ibáñez" [mailto:[EMAIL PROTECTED]
Sent: May 21, 2004 8:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors



The problem I have with one of my sites seems to have a different cause, 
as it
does not show up when errors occur, but when requests are succesful.

So far I haven't been able to reproduce the problem.

You can try to modify "new_publish" to catch exceptions:

def new_publish(request, module_name, after_list, debug=0):
id = get_ident()
Publish._requests[id] = request
try:
x = Publish.old_publish(request, module_name, after_list, debug)
except:
del Publish._requests[id]
raise
else:
try:
del Publish._requests[id]
except KeyError:
...


Though it is unlikely the leak to be there.




[EMAIL PROTECTED] wrote:

>Indeed, I was hoping the ref in the dictionary there was getting in the way
>when old_publish was raising an exception, but I tried using a
>weakref.proxy()
>instead and that didn't help.
>
>I managed to test without Localizer (It dawned on me that since I'm looking
>at errors, it really doesn't matter if I don't have Localizer installed
>since
>not having it will generate errors, which is what I want!) ... And the
>problem
>diappeared ...
>
>So it's definitely Localizer, but it's not the Globals() request patch ...
>Or
>at least not the act of patching it itself, that causes the problem.
>
>*sigh*
>J.F.
>
>-Original Message-
>From: Dieter Maurer [mailto:[EMAIL PROTECTED]
>Sent: May 20, 2004 2:58 PM
>To: [EMAIL PROTECTED]
>Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
>[EMAIL PROTECTED]; [EMAIL PROTECTED]
>Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
>caused by errors
>
>
>[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
>  
>
>>...
>>def new_publish(request, module_name, after_list, debug=0): 
>>   id = get_ident() 
>>   print "Localizer got thread id: " + str(id)
>>   Publish._requests[id] = request 
>>   print "Request dict is now: " + str(Publish._requests)   
>>   x = Publish.old_publish(request, module_name, after_list, debug) 
>>   try:
>>   del Publish._requests[id]
>>   except KeyError:
>>...
>>
>>
>
>This code cannot leak requests in large numbers (though it may
>leak a few requests for some time).
>
>Reason: Zope does not normally create new threads. This implies
>that thread_ids are reused and thereby old requests flushed from
>the dict.
>
>  
>


-- 
J. David Ibáñez
Founder and CTO of Itaapy <http://www.itaapy.com>
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Indeed, I was hoping the ref in the dictionary there was getting in the way
when old_publish was raising an exception, but I tried using a
weakref.proxy()
instead and that didn't help.

I managed to test without Localizer (It dawned on me that since I'm looking
at errors, it really doesn't matter if I don't have Localizer installed
since
not having it will generate errors, which is what I want!) ... And the
problem
diappeared ...

So it's definitely Localizer, but it's not the Globals() request patch ...
Or
at least not the act of patching it itself, that causes the problem.

*sigh*
J.F.

-Original Message-
From: Dieter Maurer [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 2:58 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors


[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
> ...
>def new_publish(request, module_name, after_list, debug=0): 
>id = get_ident() 
>print "Localizer got thread id: " + str(id)
>Publish._requests[id] = request 
>print "Request dict is now: " + str(Publish._requests)   
>x = Publish.old_publish(request, module_name, after_list, debug) 
>try:
>del Publish._requests[id]
>except KeyError:
> ...

This code cannot leak requests in large numbers (though it may
leak a few requests for some time).

Reason: Zope does not normally create new threads. This implies
that thread_ids are reused and thereby old requests flushed from
the dict.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Dieter Maurer
[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
> ...
>def new_publish(request, module_name, after_list, debug=0): 
>id = get_ident() 
>print "Localizer got thread id: " + str(id)
>Publish._requests[id] = request 
>print "Request dict is now: " + str(Publish._requests)   
>x = Publish.old_publish(request, module_name, after_list, debug) 
>try:
>del Publish._requests[id]
>except KeyError:
> ...

This code cannot leak requests in large numbers (though it may
leak a few requests for some time).

Reason: Zope does not normally create new threads. This implies
that thread_ids are reused and thereby old requests flushed from
the dict.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Juan David,

OK, 1.1.0a3 indeed suffers from the same problem, I'm going to take a closer
look the the publishing mechanism to see if I can find anything relevant.

Thanks,
J.F.

-Original Message-
From: "J. David Ibáñez" [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 10:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors



The "a" in 1.1.0a3 means "alpha". If the problem is in Localizer
1.0.1 I think it will be in 1.1.0a3 too.

Yes, I have a site with a similar behaviour, unfortunately today
the network connection is too slooow, so I can't test that site. I'm
going to do some local tests, though.

Yes, the problem might be the fact that "new_publish" does not
catch exceptions to remove the request from the dictionary.



[EMAIL PROTECTED] wrote:

>Juan David,
>
>I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...
>
>BTW, does that mean it's alpha software or something? is 1.1.0 stable?
>
>If the problem comes from Localizer I suspect this patch might be the
cause:
>
>def new_publish(request, module_name, after_list, debug=0): 
>id = get_ident() 
>print "Localizer got thread id: " + str(id)
>Publish._requests[id] = request 
>print "Request dict is now: " + str(Publish._requests)   
>x = Publish.old_publish(request, module_name, after_list, debug) 
>try:
>del Publish._requests[id]
>except KeyError:
># XXX 
># Some people has reported that sometimes a KeyError exception is
># raised in the previous line, I haven't been able to reproduce it.
># This try/except clause seems to work. I'd prefer to understand
># what is happening.
>LOG('Localizer', PROBLEM,
>"The thread number %s don't has a request object associated." %
>id)
>except:
>print "Localizer got an exception" 
> 
>return x 
>
>What happens if an exception occurs here:
>
>x = Publish.old_publish(request, module_name, after_list, debug) 
>
>??
>
>If some clean up code is supposed to occur it might be held back by the
fact
>that
>there's a reference to the request kept in the global dictionary ...
>At least that's my theory :)
>But then I'm mostly uneducated about the intricacies of exception handling
>at this
>level, though I've been forced to learn quickly :)
>
>I'd be curious to know at least if you can reproduce the leak by generating
>loads
>of errors on a Localizer enabled site?
>
>One option would also be to simply wrap it in a try:catch and see what
>happens there
>... But I noticed 1.1.0a3 uses a different model for wrapping the request,
>so that
>just might do the trick.
>
>Thanks for a very cool and useful product BTW.  This is combination with
the
>TranslationService is saving me loads of time.
>
>Cheers,
>J.F.
>
>-Original Message-
>From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
>Behalf Of "J. David Ibáñez"
>Sent: May 20, 2004 6:47 AM
>To: Tres Seaver
>Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
>[EMAIL PROTECTED]; [EMAIL PROTECTED]
>Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
>errors
>
>
>
>Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
>the module
>"ZPublisher.Publish". The keys are integers, they are the values 
>returned by Python's
>"thread.get_ident". The values are "ZPublisher.HTTPRequest.HTTPRequest"
>instances, they are not acquisition wrappers.
>
>Jean-François, which Localizer version do you use? Just to be sure we 
>look at
>the same code.
>
>
>  
>


-- 
J. David Ibáñez
Founder and CTO of Itaapy <http://www.itaapy.com>
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread J. David Ibáñez
The "a" in 1.1.0a3 means "alpha". If the problem is in Localizer
1.0.1 I think it will be in 1.1.0a3 too.
Yes, I have a site with a similar behaviour, unfortunately today
the network connection is too slooow, so I can't test that site. I'm
going to do some local tests, though.
Yes, the problem might be the fact that "new_publish" does not
catch exceptions to remove the request from the dictionary.

[EMAIL PROTECTED] wrote:
Juan David,
I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...
BTW, does that mean it's alpha software or something? is 1.1.0 stable?
If the problem comes from Localizer I suspect this patch might be the cause:
def new_publish(request, module_name, after_list, debug=0): 
   id = get_ident() 
   print "Localizer got thread id: " + str(id)
   Publish._requests[id] = request 
   print "Request dict is now: " + str(Publish._requests)   
   x = Publish.old_publish(request, module_name, after_list, debug) 
   try:
   del Publish._requests[id]
   except KeyError:
   # XXX 
   # Some people has reported that sometimes a KeyError exception is
   # raised in the previous line, I haven't been able to reproduce it.
   # This try/except clause seems to work. I'd prefer to understand
   # what is happening.
   LOG('Localizer', PROBLEM,
   "The thread number %s don't has a request object associated." %
id)
   except:
   print "Localizer got an exception" 

   return x 

What happens if an exception occurs here:
   x = Publish.old_publish(request, module_name, after_list, debug) 

??
If some clean up code is supposed to occur it might be held back by the fact
that
there's a reference to the request kept in the global dictionary ...
At least that's my theory :)
But then I'm mostly uneducated about the intricacies of exception handling
at this
level, though I've been forced to learn quickly :)
I'd be curious to know at least if you can reproduce the leak by generating
loads
of errors on a Localizer enabled site?
One option would also be to simply wrap it in a try:catch and see what
happens there
... But I noticed 1.1.0a3 uses a different model for wrapping the request,
so that
just might do the trick.
Thanks for a very cool and useful product BTW.  This is combination with the
TranslationService is saving me loads of time.
Cheers,
J.F.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of "J. David Ibáñez"
Sent: May 20, 2004 6:47 AM
To: Tres Seaver
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
errors

Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
the module
"ZPublisher.Publish". The keys are integers, they are the values 
returned by Python's
"thread.get_ident". The values are "ZPublisher.HTTPRequest.HTTPRequest"
instances, they are not acquisition wrappers.

Jean-François, which Localizer version do you use? Just to be sure we 
look at
the same code.

 


--
J. David Ibáñez
Founder and CTO of Itaapy 
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Juan David,

I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...

BTW, does that mean it's alpha software or something? is 1.1.0 stable?

If the problem comes from Localizer I suspect this patch might be the cause:

def new_publish(request, module_name, after_list, debug=0): 
id = get_ident() 
print "Localizer got thread id: " + str(id)
Publish._requests[id] = request 
print "Request dict is now: " + str(Publish._requests)   
x = Publish.old_publish(request, module_name, after_list, debug) 
try:
del Publish._requests[id]
except KeyError:
# XXX 
# Some people has reported that sometimes a KeyError exception is
# raised in the previous line, I haven't been able to reproduce it.
# This try/except clause seems to work. I'd prefer to understand
# what is happening.
LOG('Localizer', PROBLEM,
"The thread number %s don't has a request object associated." %
id)
except:
print "Localizer got an exception" 
 
return x 

What happens if an exception occurs here:

x = Publish.old_publish(request, module_name, after_list, debug) 

??

If some clean up code is supposed to occur it might be held back by the fact
that
there's a reference to the request kept in the global dictionary ...
At least that's my theory :)
But then I'm mostly uneducated about the intricacies of exception handling
at this
level, though I've been forced to learn quickly :)

I'd be curious to know at least if you can reproduce the leak by generating
loads
of errors on a Localizer enabled site?

One option would also be to simply wrap it in a try:catch and see what
happens there
... But I noticed 1.1.0a3 uses a different model for wrapping the request,
so that
just might do the trick.

Thanks for a very cool and useful product BTW.  This is combination with the
TranslationService is saving me loads of time.

Cheers,
J.F.

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of "J. David Ibáñez"
Sent: May 20, 2004 6:47 AM
To: Tres Seaver
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
errors



Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
the module
"ZPublisher.Publish". The keys are integers, they are the values 
returned by Python's
"thread.get_ident". The values are "ZPublisher.HTTPRequest.HTTPRequest"
instances, they are not acquisition wrappers.

Jean-François, which Localizer version do you use? Just to be sure we 
look at
the same code.


Tres Seaver wrote:

> [EMAIL PROTECTED] wrote:
>
>> Gents,
>>
>> In following to the discussions regarding the leakage zope.org and my 
>> site
>> suffer from, I've done some testing, and here's what I've found.
>>
>> First of all, the leak does seem to occur when errors occur.  Unlike 
>> what
>> Chris suggested, I still suffer from this leak even when I completely
>> remove standard_error_message.
>>
>> How I tested:
>>
>> I use JMeter, and a 99% identical box, dedicated for testing:
>>
>> - RH 7.3
>> - Python 2.3.3 compiled with ./configure and no options
>> - Zope 2.7 + a variety of products, both custom and well known (CMF,
>> Localizer, etc ...)
>>
>> I look at my error log on my main site to see the types of errors 
>> coming up
>> and note the URL/parameters.
>> Such errors include KeyErrors, a RecursionDepth error, a
>> Method-something-or-other error, and so on.
>>
>> I setup JMeter with multiple concurrent threads and run it.  Within 
>> minutes
>> I can see refcounts growing abnormally on the test server.
>>
>> I then shut everything down and minimize the caches.  The following 
>> are left
>> around that shouldn't be:
>>
>> Class  May 19, 2004 1:52 pm  May 19, 2004 2:26 pm  Delta  
>> Acquisition.ImplicitAcquirerWrapper  1025  1309  +284  
>> ZServer.HTTPResponse.ZServerHTTPResponse  150  247  +97  
>> ZPublisher.BaseRequest.RequestContainer  146  240  +94  
>> ZPublisher.HTTPRequest.HTTPRequest  172  262  +90
>>
>> Right now I've focused on finding out why the Requests are still around.
>>
>> Using the gc module, I've found that all leaked requests are being 
>> held by a
>> dictionary.  If I look at said dictionaries, this is what they might 
>> look
>> like:
>>
>> {'REQUEST': >
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh

>>
>> _site/francais/mapsatoz>},
>>  {'REQUEST': >
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh

>>
>> _site/francais/mapsatoz>},
>>  {'REQUEST': >
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh

>>
>> _site/english/search/political>},
>>  {'REQUEST': >
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh

>>
>> _site/english/search/political>},
>>  {'REQUEST': >
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh

>>
>> _site/english/search/political