Here is a short story about the changes we made yesterday in the
zope.security package :
Yesterday we deployed our first portal using buildout and eggs :) :)
Before we could do this we run into performance problems :( :(
On our development test instance we found that we got a 5% CPU load for
each of our remotetask threads. Because we have 5 of them running per
portal and we have 3 portals active... :(
After some investigation with the profiler :
lovely.remotetask throws an IndexError exception if there is no job to
do. It is doing this once a second.
The publisher gets a traceback for the exception using python's
traceback.extract_stack() uses python's linecache module.
Now here is the problem coming in :
linecache is extremely slow when using eggs !
We could measure 54 ms for the time spent in the publisher !
So we changed remotetask to not throw an IndexError.
But the publisher still took 27ms.
Profiling again showed us that traceback.extract_stack() was still
Finally we found it in zope.security.manager.py
newInteraction was storing a traceback to be able to print a nice
traceback in case newInteraction is called a second time. This is really
a good thing for the developer because you get a very detailed error
report which shows you exactly from where newInteraction was called the
But for which price:
Removing the extraction of the traceback put down the publisher time to
!!!!! 2ms !!!!!
So we decided to remove this feature.
The change is now in the newest egg for zope.security version 3.4.0b2
With this new version I also measured the time with zope as a trunk
checkout (no eggs involved).
The publisher is now twice as fast as it was before !
I'm writing this just to show everyone what can happen if not enough
care is taken in really critical parts inside the zope core.
newInteraction is called exactly once for each request but was taking
50% of the time (without eggs) for the publisher.
Zope3-dev mailing list