Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote:
On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones
[EMAIL PROTECTED] wrote:
I'm investigating why our product ZODB is growing when it shouldn't. I've
found that the undo log has entries in it every five minutes saying
Installed product CallProfiler by Zope. More infrequently (not
regularly) we also have Installed product ZOracleDA by Zope.
Was zope restarting (or being restarted) every five minutes?
No, but it is being pinged. By that, I mean that we have a script that tries
to access the ZEO server every five minutes to make sure it's all OK. The
script does the following::
import sys, os
sys.path.append('/app/zope/zopecontrol_code/')
sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python')
os.environ['ZEOPORT'] = '8076'
os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com'
import Zope
# now do stuff with the Zope.app() (we don't actually have to do anything
# to trigger the ZODB note generation - the import is enough)
Now, the problem lies in the way that the CallProfiler gets itself into the
Control Panel. This code was copied from the LeakFinder product, so I assume
it'll have the same problem. The code is::
def installControlPanel(context, panelClass):
from App.ApplicationManager import ApplicationManager
from Acquisition import aq_base
app = context._ProductContext__app
cp = app.Control_Panel
id = panelClass.id
if 0: # Enable to clean up the control panel.
try: del cp._objects
except: pass
cp.id # Unghostify.
if hasattr(cp, id):
return cp._getOb(id)
if cp.__dict__.has_key('_objects'):
# _objects has been overridden. We have to persist.
existing = getattr(aq_base(cp), id, None)
if existing is None or existing.__class__ != panelClass:
cp._setObject(id, panelClass())
else:
# Don't persist what we don't have to.
objects = ApplicationManager._objects
objects = filter(lambda o, id=id: o['id'] != id, objects)
ApplicationManager._objects = objects + (
{'id':id, 'meta_type':panelClass.meta_type},)
try: delattr(cp, id)
except: pass
setattr(ApplicationManager, id, panelClass())
return cp._getOb(id)
Where panelClass is my Product's class.
When the ping script imports Zope, the product is installed. If it's
installed as a regular product, using context.registerClass, all is OK. If
it is installed using the above function, I get ZODB transaction notes of the
form Installed product CallProfiler by Zope 2002-04-08 11:14:10 PM.
It's all a bit black magic for me though. I honestly don't know why half of
the function above actually exists. I'm going to keep looking though, but if
someone could help, I'd be extremely grateful.
Richard
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 9 Apr 2002 09:27, Richard Jones wrote:
On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote:
On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones
[EMAIL PROTECTED] wrote:
I'm investigating why our product ZODB is growing when it shouldn't.
I've found that the undo log has entries in it every five minutes
saying Installed product CallProfiler by Zope. More infrequently (not
regularly) we also have Installed product ZOracleDA by Zope.
Was zope restarting (or being restarted) every five minutes?
No, but it is being pinged. By that, I mean that we have a script that
tries to access the ZEO server every five minutes to make sure it's all OK.
The script does the following::
import sys, os
sys.path.append('/app/zope/zopecontrol_code/')
sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python')
os.environ['ZEOPORT'] = '8076'
os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com'
import Zope
# now do stuff with the Zope.app() (we don't actually have to do anything
# to trigger the ZODB note generation - the import is enough)
Setting the environment var ZEO_CLIENT stops the message - but I guess I'd
like to know why this product is causing the message when none of the others
are. It's all due to the code in installControlPanel, and I'd rather it not
be dodgy :)
Richard
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones [EMAIL PROTECTED] wrote: I'm investigating why our product ZODB is growing when it shouldn't. I've found that the undo log has entries in it every five minutes saying Installed product CallProfiler by Zope. More infrequently (not regularly) we also have Installed product ZOracleDA by Zope. Was zope restarting (or being restarted) every five minutes? Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Call Profiler strangeness - lots of ZODB transactions
I'm investigating why our product ZODB is growing when it shouldn't. I've found that the undo log has entries in it every five minutes saying Installed product CallProfiler by Zope. More infrequently (not regularly) we also have Installed product ZOracleDA by Zope. The Call Profiler was designed specifically not to hit the ZODB - it's in-memory only with. Our setup consists of seven ZEO clients talking to the one ZEO server - the one that's getting all these hits in the ZODB. Does anyone else see this happening? Anyone have any ideas why we're seeing this behaviour? Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Mon, 4 Feb 2002 21:42, seb bacon wrote:
On Sun, 2002-02-03 at 21:58, Richard Jones wrote:
On Fri, 1 Feb 2002 21:53, seb bacon wrote:
I wrote:
I've just announced our Call Profiler product on zope.org and the
zope announce list (waiting for people in different timezones to
authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/
for the impatient)
This is absolutely excellent, well done and thanks :) I added various
FSObjects to the profiled modules list and have been looking at my CMF
systems - it's really instructive.
Please send me the config entries for them. Also, if anyone has used ZPT
with it, let me know if my guess of the config for them is correct :)
in profileable_modules:
'FSPythonScript': Profileable('Products.CMFCore.FSPythonScript',
'FSPythonScript', '__call__'),
'FSDTMLMethod': Profileable('Products.CMFCore.FSDTMLMethod',
'FSDTMLMethod', '__call__'),
'FSPageTemplate': Profileable('Products.CMFCore.FSPageTemplate',
'FSPageTemplate', '__call__'),
Thanks!
- the trace for a complete template could be slightly clearer. The
colour coding for the start and end of a module could have a clear
legend, for example.
I'll put a legend in. Is the colour coding what you meant by clearer or
is there something else?
Yes, that's what I meant.
OK, it's done. Will be in the next release (probably today :)
Actually, another thing just occurred to me:
the meaning of the ... ellipsis is not clear - I think you need a
different symbol to emphasise the hierarchy (perhaps a vertical bar?)
OK, I'll look into it. The '...' is definitely a placeholder :)
Richard
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Fri, 1 Feb 2002 21:53, seb bacon wrote: I wrote: I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) This is absolutely excellent, well done and thanks :) I added various FSObjects to the profiled modules list and have been looking at my CMF systems - it's really instructive. Please send me the config entries for them. Also, if anyone has used ZPT with it, let me know if my guess of the config for them is correct :) On one hand, the performance hit when dynamically patching the methods is zero when the product is not active, but it does mean diddling with methods that really probably should be left alone. On the other hand, having changes to the core code to test for profiling being enabled introduces a small performance hit even when profiling is not activated. You could do this check only if zope is being run in debug mode; then there would be zero perormance hit in production sites. No, there's still the performance check for am I in debug mode? which is equivalent to am I in profiling mode?. Sure, it's small, but it's there. Any comments? I think the reporting could do with some usability improvements. It's good now, but takes a while to navigate round. Yep, I know :) - All the reports could benefit from being sortable by column. In the works already :) - the trace for a complete template could be slightly clearer. The colour coding for the start and end of a module could have a clear legend, for example. I'll put a legend in. Is the colour coding what you meant by clearer or is there something else? - there needs to be some solution for really long URLs, which require lots of horizontal scrolling - just displaying the final parts of the path, with the rest of the url hidden somehow? Yep, that's a good idea. I can't think of exactly what else I would do to improve it right now I've already got some more code in place that I'm testing right now which does aggregation of matching result profiles. That is, if a request results in the same call profile, the timings are aggregated into min/average/max. Really neat :) Thanks for the feedback! Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) This is absolutely excellent, well done and thanks :) I added various FSObjects to the profiled modules list and have been looking at my CMF systems - it's really instructive. It does pose a question though: would it be better to have support for this stuff in the core, or is it OK for a product like this to dynamically patch the call methods when it needs to? I see nothing wrong with it being a monkeypatch product. On the other hand, it is a very useful tool, and as such could benefit from being in the core simply so that it always gets maintained with the core. On one hand, the performance hit when dynamically patching the methods is zero when the product is not active, but it does mean diddling with methods that really probably should be left alone. On the other hand, having changes to the core code to test for profiling being enabled introduces a small performance hit even when profiling is not activated. You could do this check only if zope is being run in debug mode; then there would be zero perormance hit in production sites. Any comments? I think the reporting could do with some usability improvements. It's good now, but takes a while to navigate round. - All the reports could benefit from being sortable by column. - the trace for a complete template could be slightly clearer. The colour coding for the start and end of a module could have a clear legend, for example. - there needs to be some solution for really long URLs, which require lots of horizontal scrolling - just displaying the final parts of the path, with the rest of the url hidden somehow? I can't think of exactly what else I would do to improve it right now, though. And that's the kind of change I never get round to bothering with, personally... Sorry these aren't really constructive ideas, but you did ask for feedback :-) seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On 1/2/02 1:13 am, Richard Jones [EMAIL PROTECTED] wrote: I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) [snip] Any comments? I've been using it for *5* minutes and have found a method that was taking 95%! of one of our most often used scripts and another python script that was effectively stalling the servergobsmacked. Huge kudos and thanks to Richard!! :) Tone. -- Dr Tony McDonald, Assistant Director, FMCC, http://www.fmcc.org.uk/ The Medical School, Newcastle University Tel: +44 191 243 6140 A Zope list for UK HE/FE http://www.fmcc.org.uk/mailman/listinfo/zope ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
Doesnt seem to like being refreshed and some event items arent in the time dict, I noticed these were items that I had got in a ram cache. So i just stuck an if at line 496. Other than that its a great product. Most important tool since refresh if you ask me... - Original Message - From: Richard Jones [EMAIL PROTECTED] To: zope-dev [EMAIL PROTECTED] Sent: Thursday, January 31, 2002 5:13 PM Subject: [Zope-dev] Call Profiler I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) In a nutshell, it patches the rendering guts of DTMLMethod, ZSQL, etc. and the ZPublisher so that we can get timing marks for requests as the calls are made. The patching is quite simple (once I got the hang of it ;) and reversible, and means that there's no changes required to the Zope core to enable the testing (which our initial versions did). It does pose a question though: would it be better to have support for this stuff in the core, or is it OK for a product like this to dynamically patch the call methods when it needs to? On one hand, the performance hit when dynamically patching the methods is zero when the product is not active, but it does mean diddling with methods that really probably should be left alone. On the other hand, having changes to the core code to test for profiling being enabled introduces a small performance hit even when profiling is not activated. Any comments? Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Call Profiler
I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) In a nutshell, it patches the rendering guts of DTMLMethod, ZSQL, etc. and the ZPublisher so that we can get timing marks for requests as the calls are made. The patching is quite simple (once I got the hang of it ;) and reversible, and means that there's no changes required to the Zope core to enable the testing (which our initial versions did). It does pose a question though: would it be better to have support for this stuff in the core, or is it OK for a product like this to dynamically patch the call methods when it needs to? On one hand, the performance hit when dynamically patching the methods is zero when the product is not active, but it does mean diddling with methods that really probably should be left alone. On the other hand, having changes to the core code to test for profiling being enabled introduces a small performance hit even when profiling is not activated. Any comments? Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
Richard Jones wrote I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) From my point of view, this is a critical piece for Zope - right now, trying to work out what's going on in a Zope page render is somewhat painful. We found once we'd turned this onto our production code (and Richard had made my initial vile hack actually work in a way that didn't cause bleeding from the brain) we were able to very very quickly spot a whole lot of stuff that was Just Not Doing The Right Thing. Unfortunately, the existing python profiling stuff is at too low a level to be useful - this stuff really fills a need. Plus, if it's integrated into Zope, Richard doesn't have to spend work time on maintaining it :) Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Fri, 1 Feb 2002 14:10, Anthony Baxter wrote: Richard Jones wrote I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) ... Richard had made my initial vile hack actually work in a way that didn't cause bleeding from the brain) No comment ;) Unfortunately, the existing python profiling stuff is at too low a level to be useful - this stuff really fills a need. Although being able to tie this into the python profiler would _rock_ in terms of profiling websites that use a mix of DTML/ZPT and Python products. Even it it's just to get the timing marks where the higher level stuff like DTML/ZPT calls out to the python code (ie. getting timing for the python calls at the highest level) Plus, if it's integrated into Zope, Richard doesn't have to spend work time on maintaining it :) But I do have time for screenshots - there's some up on the product page now for those who have a short curiosity span :) Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
This is really cool. The implementation looks pretty reasonable too! Definitely useful for folks doing development and wondering why their scripts take 2 secs to run! ;-) I think this will be in many folks' toolbags soon. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
