Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Dieter Maurer
[EMAIL PROTECTED] wrote at 2004-5-19 14:44 -0400:
 ...
In following to the discussions regarding the leakage zope.org and my site
suffer from, I've done some testing, and here's what I've found.

First of all, the leak does seem to occur when errors occur.  Unlike what
Chris suggested, I still suffer from this leak even when I completely
remove standard_error_message.

I tried to reproduce this behaviour -- but failed.

  Despite errors and a standard standard_error_meesage
  nothing leaked...

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Dieter Maurer
[EMAIL PROTECTED] wrote at 2004-5-19 14:44 -0400:
 ...
Right now I've focused on finding out why the Requests are still around.

Using the gc module, I've found that all leaked requests are being held by a
dictionary.  If I look at said dictionaries, this is what they might look
like:

{'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/francais/mapsatoz},
 {'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/francais/mapsatoz},

These dictionaries look like the instance dicts of
the ZPublisher.BaseRequest.RequestContainers.

When you have any cycle involving an AcquisitionWrapper,
then its reference to the RequestContainer will keep this alive
together with its instance dict.

You may analyse the content of your requests.
Do they contain acquisition wrapped objects (or other objects
with (maybe indirect) references to this dict)?

If not, REQUEST does not seem to be the root of your cycles
but rather a side effect.


More important notes:

  *  When you see AcquisitionWrappers in the reference information
 in Control_Panel -- Debug Information, then
 there *must* also be other prominent ExtensionClass'es --
 the aq_base of these wrappers.

 These classes may provide a valuable clue towards the cause
 of the cycles -- of global variables (by the way).

  *  The reference information in Control_Panel covers only
 instance objects but not elementary Python data types.

 Thus, when cycles are caused by elementary Python objects,
 we may not directly see them in the list.

 GC's DEBUG_LEAK may provide some information.

  *  The leaks may not be caused by cycles at all (though it is
 the most likely cause) but could result from a global
 cache containing references to acquisition wrapped objects.

 Such a cache should never exist -- global variables and
 class attributes must never directly or indirectly reference
 persistent objects -- as this results in nasty persistency bugs.

 Nevertheless, sometimes software is buggy. I noticed
 such a cache in Archetypes 1.3a2 (fixed in newer versions).


Said dictionary does not seem to be referred to by anything, at least not
anything the gc module can find (gc.get_referrers()).

That's probably because RequestContainer is an ExtensionClass
and ExtensionClasses are not yet (only from Zope 2.8 on) GC aware.

 ...
Also I
understand zope.org suffers from this? :)

Leaks can have innumerable causes. It is by no means clear
that your leak and the one on Zope.org has the same cause.

Based on this information also, I have no reason to believe this is caused
by product code ... I have to guess it's a Zope bug?

I do not observe leaks in my Zope installation -- even when
I make an error stress test.

This does not mean it could not be a Zope bug --
but it implies that such a bug does not surface regularly, at most
under special conditions.

In some ways this is motivating me to reduce the number of errors that could
occur, but because ALL errors seems to cause the problem, I ultimately have
no control over it and under certain circumstances I suffer heavily from
this.  404's alone probably cause this (I'm plannign to test that
explicitely as well).

It does not here

Something is special with your installation -- maybe Localizer as
Tres suggested, maybe another misbehaving product.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Andreas Jung
I have not followed the discussion but is this leak as serious as it should 
be resolved
before the next 2.7.1 beta1 release?

Andreas
--On Donnerstag, 20. Mai 2004 9:52 Uhr +0200 Dieter Maurer 
[EMAIL PROTECTED] wrote:

[EMAIL PROTECTED] wrote at 2004-5-19 14:44 -0400:
...
Right now I've focused on finding out why the Requests are still around.
Using the gc module, I've found that all leaked requests are being held
by a dictionary.  If I look at said dictionaries, this is what they
might look like:
{'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot
/_vh _site/francais/mapsatoz},
{'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot
/_vh _site/francais/mapsatoz},
These dictionaries look like the instance dicts of
the ZPublisher.BaseRequest.RequestContainers.
When you have any cycle involving an AcquisitionWrapper,
then its reference to the RequestContainer will keep this alive
together with its instance dict.
You may analyse the content of your requests.
Do they contain acquisition wrapped objects (or other objects
with (maybe indirect) references to this dict)?
If not, REQUEST does not seem to be the root of your cycles
but rather a side effect.
More important notes:
  *  When you see AcquisitionWrappers in the reference information
 in Control_Panel -- Debug Information, then
 there *must* also be other prominent ExtensionClass'es --
 the aq_base of these wrappers.
 These classes may provide a valuable clue towards the cause
 of the cycles -- of global variables (by the way).
  *  The reference information in Control_Panel covers only
 instance objects but not elementary Python data types.
 Thus, when cycles are caused by elementary Python objects,
 we may not directly see them in the list.
 GC's DEBUG_LEAK may provide some information.
  *  The leaks may not be caused by cycles at all (though it is
 the most likely cause) but could result from a global
 cache containing references to acquisition wrapped objects.
 Such a cache should never exist -- global variables and
 class attributes must never directly or indirectly reference
 persistent objects -- as this results in nasty persistency bugs.
 Nevertheless, sometimes software is buggy. I noticed
 such a cache in Archetypes 1.3a2 (fixed in newer versions).

Said dictionary does not seem to be referred to by anything, at least not
anything the gc module can find (gc.get_referrers()).
That's probably because RequestContainer is an ExtensionClass
and ExtensionClasses are not yet (only from Zope 2.8 on) GC aware.
...
Also I
understand zope.org suffers from this? :)
Leaks can have innumerable causes. It is by no means clear
that your leak and the one on Zope.org has the same cause.
Based on this information also, I have no reason to believe this is
caused by product code ... I have to guess it's a Zope bug?
I do not observe leaks in my Zope installation -- even when
I make an error stress test.
This does not mean it could not be a Zope bug --
but it implies that such a bug does not surface regularly, at most
under special conditions.
In some ways this is motivating me to reduce the number of errors that
could occur, but because ALL errors seems to cause the problem, I
ultimately have no control over it and under certain circumstances I
suffer heavily from this.  404's alone probably cause this (I'm plannign
to test that explicitely as well).
It does not here
Something is special with your installation -- maybe Localizer as
Tres suggested, maybe another misbehaving product.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZPT] RFC: TALES adapters and TAL/Tales variable namespaces

2004-05-20 Thread Jim Fulton
Evan Simpson wrote:
Jim Fulton wrote:
I've posted two proposals:
  http://dev.zope.org/Zope3/TALESPathExpressionAdapters
Proposes a mechanism for easily using adapters in TALES expressions.

I'm not at all clear on how the proposed mechanism is superior to the 
implementation of path segment prefixes that exists in Zope 2.   This
differs from the namespace proposal that you cite, in that the object 
named on the left of the colon is not a namespace or adapter.  It is 
simply a registered name that is associated with code that interprets 
the text to the right of the colon.  This allows constuctions such as 
a/sequence/index:2 (a.sequence[2]), a/bag/var:x/call: (a.bag[x]()), 
and an/object/adapt:foo.bar (foo.bar(an.object)).
I thought you had proposed this, but I couldn't find a proposal
or documentation.
You say this is implemented in Zope 2? Where is it documented?
Jim
--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Read-only root database doesn't work ... bug or feature?

2004-05-20 Thread robert rottermann
Paul Winkler wrote:
I'm trying to figure out how to mount my main storage read-only
with zope 2.7.0.  I'm starting to suspect that it's impossible.
I tried a few things below
-
ATTEMPT #1.
I find this in the zope.conf examples:
# Directive: read-only-database
#
# Description:
# This causes the main Zope FileStorage-backed ZODB to be opened in
# read-only mode.
#
# Default: off
#
# Example:
#
# read-only-database on
... so I uncomment that line and restart.
Zope starts OK but the database is evidently still writeable, I can
still change anything.
Is this a bug in the zope.conf examples, or a bug in zope?
-
ATTEMPT #2.
Apparently there is another read-only flag within each database config
section. So, I try that. In my zope.conf:
zodb_db main
   # Main FileStorage database
   cache-size 2
   mount-point /
   filestorage
 path $INSTANCE/var/Data.fs
 read-only on
   /filestorage
/zodb_db
When I start with this config, zope dies during product initialization
which apparently wants to commit:
--
2004-05-19T19:08:10 ERROR(200) Zope Couldn't install Formulator
Traceback (most recent call last):
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 785, in install_product
   get_transaction().commit()
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Transaction.py, line 232, in commit
   self._commit_begin(jars, subjars, subtransaction)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Transaction.py, line 340, in _commit_begin
   jar.tpc_begin(self)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Connection.py, line 692, in tpc_begin
   self._storage.tpc_begin(transaction)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/BaseStorage.py, line 142, in tpc_begin
   raise POSException.ReadOnlyError()
ReadOnlyError
Traceback (most recent call last):
 File /home/pw/Zope-2.7.0/lib/python/Zope/Startup/run.py, line 49, in ?
   run()
 File /home/pw/Zope-2.7.0/lib/python/Zope/Startup/run.py, line 19, in run
   start_zope(opts.configroot)
 File /home/pw/Zope-2.7.0/lib/python/Zope/Startup/__init__.py, line 51, in start_zope
   starter.startZope()
 File /home/pw/Zope-2.7.0/lib/python/Zope/Startup/__init__.py, line 230, in startZope
   Zope.startup()
 File /home/pw/Zope-2.7.0/lib/python/Zope/__init__.py, line 46, in startup
   _startup()
 File /home/pw/Zope-2.7.0/lib/python/Zope/App/startup.py, line 93, in startup
   OFS.Application.initialize(application)
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 279, in initialize
   initializer.initialize()
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 306, in initialize
   self.install_products() 
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 553, in install_products
   return install_products(app)
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 584, in install_products
   folder_permissions, raise_exc=debug_mode)
 File /home/pw/Zope-2.7.0/lib/python/OFS/Application.py, line 785, in install_product
   get_transaction().commit()
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Transaction.py, line 232, in commit
   self._commit_begin(jars, subjars, subtransaction)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Transaction.py, line 340, in _commit_begin
   jar.tpc_begin(self)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/Connection.py, line 692, in tpc_begin
   self._storage.tpc_begin(transaction)
 File /home/pw/Zope-2.7.0/lib/python/ZODB/BaseStorage.py, line 142, in tpc_begin
   raise POSException.ReadOnlyError()
ZODB.POSException.ReadOnlyError

-
So, am I just plain S.O.L.?  Is this impossible?
(Note: a similar message was sent earlier to dirstorage-users, but
experimentation has shown me that the storage implementation
doesn't seem to matter... i get the same result with filestorage,
directorystorage, or clientstorage.)
 

we had a simmilar problem. Eventually we spved it by patching zope.
However I am not sure whether the problem  was caused by Products (we do 
it for a plone site ) or by Zope itself.

what we changed is on a windows box that is not running under linux. 
thats why I do not include it here. However I can rboot that machine and 
send it to you if you like.
Robert

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by e rrors

2004-05-20 Thread Jean-Francois . Doyon
Dieter,

Thanks yet again for all the valuable input.

I've started tracking down/running into the whole ExtensionClasses
instances/issues.

As you suggest, the dict is probably help by BaseRequest.RequestContainer,
which I've
seen grow in parallel to the HTTPRequest refcounts ... Which seems to make
sense.

And as you suggest, as I've been doing this, I've been thinking I'm really
looking forward to getting ri dof ExtensionClass, makes this kind of work
much easier.

The requests themselves look clean, but if they contain ExtensionClasses
based instances
gc wouldn't find them for the same reason.  I've gotten far enough to be
able to tell
there's a variety of them.  I suspect the instance the error occured ON is
what's leaked
(So if it's a CMFDocument, then that said Document ends up leaked).  I was
just getting
to this yesterday, hopefully I'll solve that today.

As Tres suggested, I'm going to go back to looking at Localizer ... It's the
likliest suspect
at this time.  It's just hard for to test since pretty much everything I
have depends on
i18n PT's, and in some cases I have gettext() calls in Python code.
Localizer wraps
the Publish.publish, and I'm thinking that when exceptions occur in there
Localizer
needs to be aware of it and react accordingly, which it doesn't right now.

The hunt goes on ...

Cheers,
J.F.

-Original Message-
From: Dieter Maurer [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 3:52 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by
errors


[EMAIL PROTECTED] wrote at 2004-5-19 14:44 -0400:
 ...
Right now I've focused on finding out why the Requests are still around.

Using the gc module, I've found that all leaked requests are being held by
a
dictionary.  If I look at said dictionaries, this is what they might look
like:

{'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v
h
_site/francais/mapsatoz},
 {'REQUEST': HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_v
h
_site/francais/mapsatoz},

These dictionaries look like the instance dicts of
the ZPublisher.BaseRequest.RequestContainers.

When you have any cycle involving an AcquisitionWrapper,
then its reference to the RequestContainer will keep this alive
together with its instance dict.

You may analyse the content of your requests.
Do they contain acquisition wrapped objects (or other objects
with (maybe indirect) references to this dict)?

If not, REQUEST does not seem to be the root of your cycles
but rather a side effect.


More important notes:

  *  When you see AcquisitionWrappers in the reference information
 in Control_Panel -- Debug Information, then
 there *must* also be other prominent ExtensionClass'es --
 the aq_base of these wrappers.

 These classes may provide a valuable clue towards the cause
 of the cycles -- of global variables (by the way).

  *  The reference information in Control_Panel covers only
 instance objects but not elementary Python data types.

 Thus, when cycles are caused by elementary Python objects,
 we may not directly see them in the list.

 GC's DEBUG_LEAK may provide some information.

  *  The leaks may not be caused by cycles at all (though it is
 the most likely cause) but could result from a global
 cache containing references to acquisition wrapped objects.

 Such a cache should never exist -- global variables and
 class attributes must never directly or indirectly reference
 persistent objects -- as this results in nasty persistency bugs.

 Nevertheless, sometimes software is buggy. I noticed
 such a cache in Archetypes 1.3a2 (fixed in newer versions).


Said dictionary does not seem to be referred to by anything, at least not
anything the gc module can find (gc.get_referrers()).

That's probably because RequestContainer is an ExtensionClass
and ExtensionClasses are not yet (only from Zope 2.8 on) GC aware.

 ...
Also I
understand zope.org suffers from this? :)

Leaks can have innumerable causes. It is by no means clear
that your leak and the one on Zope.org has the same cause.

Based on this information also, I have no reason to believe this is caused
by product code ... I have to guess it's a Zope bug?

I do not observe leaks in my Zope installation -- even when
I make an error stress test.

This does not mean it could not be a Zope bug --
but it implies that such a bug does not surface regularly, at most
under special conditions.

In some ways this is motivating me to reduce the number of errors that
could
occur, but because ALL errors seems to cause the problem, I ultimately have
no control over it and under certain circumstances I suffer heavily from
this.  404's alone probably cause this (I'm plannign to test that
explicitely as well).

It does not here

Something is special with your 

RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Juan David,

I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...

BTW, does that mean it's alpha software or something? is 1.1.0 stable?

If the problem comes from Localizer I suspect this patch might be the cause:

def new_publish(request, module_name, after_list, debug=0): 
id = get_ident() 
print Localizer got thread id:  + str(id)
Publish._requests[id] = request 
print Request dict is now:  + str(Publish._requests)   
x = Publish.old_publish(request, module_name, after_list, debug) 
try:
del Publish._requests[id]
except KeyError:
# XXX 
# Some people has reported that sometimes a KeyError exception is
# raised in the previous line, I haven't been able to reproduce it.
# This try/except clause seems to work. I'd prefer to understand
# what is happening.
LOG('Localizer', PROBLEM,
The thread number %s don't has a request object associated. %
id)
except:
print Localizer got an exception 
 
return x 

What happens if an exception occurs here:

x = Publish.old_publish(request, module_name, after_list, debug) 

??

If some clean up code is supposed to occur it might be held back by the fact
that
there's a reference to the request kept in the global dictionary ...
At least that's my theory :)
But then I'm mostly uneducated about the intricacies of exception handling
at this
level, though I've been forced to learn quickly :)

I'd be curious to know at least if you can reproduce the leak by generating
loads
of errors on a Localizer enabled site?

One option would also be to simply wrap it in a try:catch and see what
happens there
... But I noticed 1.1.0a3 uses a different model for wrapping the request,
so that
just might do the trick.

Thanks for a very cool and useful product BTW.  This is combination with the
TranslationService is saving me loads of time.

Cheers,
J.F.

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of J. David Ibáñez
Sent: May 20, 2004 6:47 AM
To: Tres Seaver
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
errors



Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
the module
ZPublisher.Publish. The keys are integers, they are the values 
returned by Python's
thread.get_ident. The values are ZPublisher.HTTPRequest.HTTPRequest
instances, they are not acquisition wrappers.

Jean-François, which Localizer version do you use? Just to be sure we 
look at
the same code.


Tres Seaver wrote:

 [EMAIL PROTECTED] wrote:

 Gents,

 In following to the discussions regarding the leakage zope.org and my 
 site
 suffer from, I've done some testing, and here's what I've found.

 First of all, the leak does seem to occur when errors occur.  Unlike 
 what
 Chris suggested, I still suffer from this leak even when I completely
 remove standard_error_message.

 How I tested:

 I use JMeter, and a 99% identical box, dedicated for testing:

 - RH 7.3
 - Python 2.3.3 compiled with ./configure and no options
 - Zope 2.7 + a variety of products, both custom and well known (CMF,
 Localizer, etc ...)

 I look at my error log on my main site to see the types of errors 
 coming up
 and note the URL/parameters.
 Such errors include KeyErrors, a RecursionDepth error, a
 Method-something-or-other error, and so on.

 I setup JMeter with multiple concurrent threads and run it.  Within 
 minutes
 I can see refcounts growing abnormally on the test server.

 I then shut everything down and minimize the caches.  The following 
 are left
 around that shouldn't be:

 Class  May 19, 2004 1:52 pm  May 19, 2004 2:26 pm  Delta  
 Acquisition.ImplicitAcquirerWrapper  1025  1309  +284  
 ZServer.HTTPResponse.ZServerHTTPResponse  150  247  +97  
 ZPublisher.BaseRequest.RequestContainer  146  240  +94  
 ZPublisher.HTTPRequest.HTTPRequest  172  262  +90

 Right now I've focused on finding out why the Requests are still around.

 Using the gc module, I've found that all leaked requests are being 
 held by a
 dictionary.  If I look at said dictionaries, this is what they might 
 look
 like:

 {'REQUEST': HTTPRequest,

URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh


 _site/francais/mapsatoz},
  {'REQUEST': HTTPRequest,

URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh


 _site/francais/mapsatoz},
  {'REQUEST': HTTPRequest,

URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh


 _site/english/search/political},
  {'REQUEST': HTTPRequest,

URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh


 _site/english/search/political},
  {'REQUEST': HTTPRequest,

URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh


 _site/english/search/political},
  {'REQUEST': HTTPRequest,


Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread J. David Ibáñez
The a in 1.1.0a3 means alpha. If the problem is in Localizer
1.0.1 I think it will be in 1.1.0a3 too.
Yes, I have a site with a similar behaviour, unfortunately today
the network connection is too slooow, so I can't test that site. I'm
going to do some local tests, though.
Yes, the problem might be the fact that new_publish does not
catch exceptions to remove the request from the dictionary.

[EMAIL PROTECTED] wrote:
Juan David,
I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...
BTW, does that mean it's alpha software or something? is 1.1.0 stable?
If the problem comes from Localizer I suspect this patch might be the cause:
def new_publish(request, module_name, after_list, debug=0): 
   id = get_ident() 
   print Localizer got thread id:  + str(id)
   Publish._requests[id] = request 
   print Request dict is now:  + str(Publish._requests)   
   x = Publish.old_publish(request, module_name, after_list, debug) 
   try:
   del Publish._requests[id]
   except KeyError:
   # XXX 
   # Some people has reported that sometimes a KeyError exception is
   # raised in the previous line, I haven't been able to reproduce it.
   # This try/except clause seems to work. I'd prefer to understand
   # what is happening.
   LOG('Localizer', PROBLEM,
   The thread number %s don't has a request object associated. %
id)
   except:
   print Localizer got an exception 

   return x 

What happens if an exception occurs here:
   x = Publish.old_publish(request, module_name, after_list, debug) 

??
If some clean up code is supposed to occur it might be held back by the fact
that
there's a reference to the request kept in the global dictionary ...
At least that's my theory :)
But then I'm mostly uneducated about the intricacies of exception handling
at this
level, though I've been forced to learn quickly :)
I'd be curious to know at least if you can reproduce the leak by generating
loads
of errors on a Localizer enabled site?
One option would also be to simply wrap it in a try:catch and see what
happens there
... But I noticed 1.1.0a3 uses a different model for wrapping the request,
so that
just might do the trick.
Thanks for a very cool and useful product BTW.  This is combination with the
TranslationService is saving me loads of time.
Cheers,
J.F.
-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of J. David Ibáñez
Sent: May 20, 2004 6:47 AM
To: Tres Seaver
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
errors

Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
the module
ZPublisher.Publish. The keys are integers, they are the values 
returned by Python's
thread.get_ident. The values are ZPublisher.HTTPRequest.HTTPRequest
instances, they are not acquisition wrappers.

Jean-François, which Localizer version do you use? Just to be sure we 
look at
the same code.

 


--
J. David Ibáñez
Founder and CTO of Itaapy http://www.itaapy.com
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Read-only root database doesn't work ... bug or feature?

2004-05-20 Thread Paul Winkler
On Thu, May 20, 2004 at 10:53:35AM +0200, robert rottermann wrote:
 Paul Winkler wrote:
 
 I'm trying to figure out how to mount my main storage read-only
 with zope 2.7.0.  I'm starting to suspect that it's impossible.
(snip)
 we had a simmilar problem. Eventually we spved it by patching zope.
 However I am not sure whether the problem  was caused by Products (we do 
 it for a plone site ) or by Zope itself.
 
 what we changed is on a windows box that is not running under linux. 
 thats why I do not include it here. However I can rboot that machine and 
 send it to you if you like.

Please do! That would be much appreciated.
Thanks!

-- 

Paul Winkler
http://www.slinkp.com

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Juan David,

OK, 1.1.0a3 indeed suffers from the same problem, I'm going to take a closer
look the the publishing mechanism to see if I can find anything relevant.

Thanks,
J.F.

-Original Message-
From: J. David Ibáñez [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 10:17 AM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]
Subject: Re: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors



The a in 1.1.0a3 means alpha. If the problem is in Localizer
1.0.1 I think it will be in 1.1.0a3 too.

Yes, I have a site with a similar behaviour, unfortunately today
the network connection is too slooow, so I can't test that site. I'm
going to do some local tests, though.

Yes, the problem might be the fact that new_publish does not
catch exceptions to remove the request from the dictionary.



[EMAIL PROTECTED] wrote:

Juan David,

I'm using 1.0.1 ... I'm just abotu to try and test with 1.1.0a3 ...

BTW, does that mean it's alpha software or something? is 1.1.0 stable?

If the problem comes from Localizer I suspect this patch might be the
cause:

def new_publish(request, module_name, after_list, debug=0): 
id = get_ident() 
print Localizer got thread id:  + str(id)
Publish._requests[id] = request 
print Request dict is now:  + str(Publish._requests)   
x = Publish.old_publish(request, module_name, after_list, debug) 
try:
del Publish._requests[id]
except KeyError:
# XXX 
# Some people has reported that sometimes a KeyError exception is
# raised in the previous line, I haven't been able to reproduce it.
# This try/except clause seems to work. I'd prefer to understand
# what is happening.
LOG('Localizer', PROBLEM,
The thread number %s don't has a request object associated. %
id)
except:
print Localizer got an exception 
 
return x 

What happens if an exception occurs here:

x = Publish.old_publish(request, module_name, after_list, debug) 

??

If some clean up code is supposed to occur it might be held back by the
fact
that
there's a reference to the request kept in the global dictionary ...
At least that's my theory :)
But then I'm mostly uneducated about the intricacies of exception handling
at this
level, though I've been forced to learn quickly :)

I'd be curious to know at least if you can reproduce the leak by generating
loads
of errors on a Localizer enabled site?

One option would also be to simply wrap it in a try:catch and see what
happens there
... But I noticed 1.1.0a3 uses a different model for wrapping the request,
so that
just might do the trick.

Thanks for a very cool and useful product BTW.  This is combination with
the
TranslationService is saving me loads of time.

Cheers,
J.F.

-Original Message-
From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]
Behalf Of J. David Ibáñez
Sent: May 20, 2004 6:47 AM
To: Tres Seaver
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by
errors



Localizer (1.0.1) dynamically patches Zope, it stores a dictionary in 
the module
ZPublisher.Publish. The keys are integers, they are the values 
returned by Python's
thread.get_ident. The values are ZPublisher.HTTPRequest.HTTPRequest
instances, they are not acquisition wrappers.

Jean-François, which Localizer version do you use? Just to be sure we 
look at
the same code.


  



-- 
J. David Ibáñez
Founder and CTO of Itaapy http://www.itaapy.com
9 rue Darwin, 75018 Paris
Tel +33 (0)1 42 23 67 45 / Fax +33 (0)1 53 28 27 88 

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZPT] RFC: TALES adapters and TAL/Tales variable namespaces

2004-05-20 Thread Evan Simpson
Jim Fulton wrote:
I thought you had proposed this, but I couldn't find a proposal
or documentation.
You say this is implemented in Zope 2? Where is it documented?
Actually, it's only implemented on evan-pathprefix-branch.  It has been 
quite a while since I had a chance to work on it, so I had forgotten 
that it was never merged, or properly documented for that matter. 
Pretty much all the information about it that isn't in the source code 
is in the ZPT list archives from August and September (e.g. 
http://mail.zope.org/pipermail/zpt/2003-August/004902.html)  So, no 
worries if you decide to take a completely different direction -- it was 
never released.

Cheers,
Evan
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZPT] RFC: TALES adapters and TAL/Tales variable namespaces

2004-05-20 Thread Jim Fulton
Evan Simpson wrote:
Jim Fulton wrote:
I thought you had proposed this, but I couldn't find a proposal
or documentation.
You say this is implemented in Zope 2? Where is it documented?

Actually, it's only implemented on evan-pathprefix-branch.  It has been 
quite a while since I had a chance to work on it, so I had forgotten 
that it was never merged, or properly documented for that matter. Pretty 
much all the information about it that isn't in the source code is in 
the ZPT list archives from August and September (e.g. 
http://mail.zope.org/pipermail/zpt/2003-August/004902.html)
Thanks for the pointer.
 So, no
worries if you decide to take a completely different direction -- it was 
never released.
I had forgotten that there was already a notion of namespaces (local and global)
and a syntax for defining variables in those namespaces.  This is very helpful.
So, we could define an adapter or prefix namespace without providing a
generalizd namespace mechanism.
I don't inderstand what motivated the special expression types for this
namespace.  For example, to get at file-system based code, couldn't you have
used the modules variable?
Jim
--
Jim Fulton   mailto:[EMAIL PROTECTED]   Python Powered!
CTO  (540) 361-1714http://www.python.org
Zope Corporation http://www.zope.com   http://www.zope.org
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


[Zope-dev] Re: [ZPT] RFC: TALES adapters and TAL/Tales variable namespaces

2004-05-20 Thread Evan Simpson
Jim Fulton wrote:
I don't inderstand what motivated the special expression types for this
namespace.  For example, to get at file-system based code, couldn't you 
have used the modules variable?
I started there, but went with the special expression types because the 
set of things that are valid to define as prefixes is completely 
disjoint from those that you would use in other TAL statements.  I've 
grown wary of tying abstract object names to Python package and module 
structure.  Also, while I find it difficult to describe precisely, I 
have a strong intuition that arbitrary objects should not be available 
for use as prefixes.

Of course, the idea of prefixes as Zope 3 adapters intead of custom-made 
prefix implementations may change things.

Cheers,
Evan @ 4-am
___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Dieter Maurer
[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
 ...
def new_publish(request, module_name, after_list, debug=0): 
id = get_ident() 
print Localizer got thread id:  + str(id)
Publish._requests[id] = request 
print Request dict is now:  + str(Publish._requests)   
x = Publish.old_publish(request, module_name, after_list, debug) 
try:
del Publish._requests[id]
except KeyError:
 ...

This code cannot leak requests in large numbers (though it may
leak a few requests for some time).

Reason: Zope does not normally create new threads. This implies
that thread_ids are reused and thereby old requests flushed from
the dict.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


Re: [Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Dieter Maurer
Andreas Jung wrote at 2004-5-20 10:02 +0200:
I have not followed the discussion but is this leak as serious as it should 
be resolved
before the next 2.7.1 beta1 release?

I doubt that it is a Zope bug...
I am unable to reproduce any leak in connection with error handling.

Otherwise, we can only answer your question when we know the reason
for the leakage.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )


RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

2004-05-20 Thread Jean-Francois . Doyon
Indeed, I was hoping the ref in the dictionary there was getting in the way
when old_publish was raising an exception, but I tried using a
weakref.proxy()
instead and that didn't help.

I managed to test without Localizer (It dawned on me that since I'm looking
at errors, it really doesn't matter if I don't have Localizer installed
since
not having it will generate errors, which is what I want!) ... And the
problem
diappeared ...

So it's definitely Localizer, but it's not the Globals() request patch ...
Or
at least not the act of patching it itself, that causes the problem.

*sigh*
J.F.

-Original Message-
From: Dieter Maurer [mailto:[EMAIL PROTECTED]
Sent: May 20, 2004 2:58 PM
To: [EMAIL PROTECTED]
Cc: [EMAIL PROTECTED]; [EMAIL PROTECTED]; [EMAIL PROTECTED];
[EMAIL PROTECTED]; [EMAIL PROTECTED]
Subject: RE: [Zope-dev] Re: Preliminary findings: Zope 2.7 leakage
caused by errors


[EMAIL PROTECTED] wrote at 2004-5-20 09:58 -0400:
 ...
def new_publish(request, module_name, after_list, debug=0): 
id = get_ident() 
print Localizer got thread id:  + str(id)
Publish._requests[id] = request 
print Request dict is now:  + str(Publish._requests)   
x = Publish.old_publish(request, module_name, after_list, debug) 
try:
del Publish._requests[id]
except KeyError:
 ...

This code cannot leak requests in large numbers (though it may
leak a few requests for some time).

Reason: Zope does not normally create new threads. This implies
that thread_ids are reused and thereby old requests flushed from
the dict.

-- 
Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://mail.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope )