Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 9 Apr 2002 09:27, Richard Jones wrote: > On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote: > > On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones > > > > <[EMAIL PROTECTED]> wrote: > > >I'm investigating why our product ZODB is growing when it shouldn't. > > > I've found that the undo log has entries in it every five minutes > > > saying "Installed product CallProfiler by Zope". More infrequently (not > > > regularly) we also have "Installed product ZOracleDA by Zope". > > > > Was zope restarting (or being restarted) every five minutes? > > No, but it is being pinged. By that, I mean that we have a script that > tries to access the ZEO server every five minutes to make sure it's all OK. > The script does the following:: > > import sys, os > sys.path.append('/app/zope/zopecontrol_code/') > sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python') > os.environ['ZEOPORT'] = '8076' > os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com' > import Zope > # now do stuff with the Zope.app() (we don't actually have to do anything > # to trigger the ZODB note generation - the import is enough) Setting the environment var ZEO_CLIENT stops the message - but I guess I'd like to know why this product is causing the message when none of the others are. It's all due to the code in installControlPanel, and I'd rather it not be dodgy :) Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote: > On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones > > <[EMAIL PROTECTED]> wrote: > >I'm investigating why our product ZODB is growing when it shouldn't. I've > >found that the undo log has entries in it every five minutes saying > >"Installed product CallProfiler by Zope". More infrequently (not > > regularly) we also have "Installed product ZOracleDA by Zope". > > Was zope restarting (or being restarted) every five minutes? No, but it is being pinged. By that, I mean that we have a script that tries to access the ZEO server every five minutes to make sure it's all OK. The script does the following:: import sys, os sys.path.append('/app/zope/zopecontrol_code/') sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python') os.environ['ZEOPORT'] = '8076' os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com' import Zope # now do stuff with the Zope.app() (we don't actually have to do anything # to trigger the ZODB note generation - the import is enough) Now, the problem lies in the way that the CallProfiler gets itself into the Control Panel. This code was copied from the LeakFinder product, so I assume it'll have the same problem. The code is:: def installControlPanel(context, panelClass): from App.ApplicationManager import ApplicationManager from Acquisition import aq_base app = context._ProductContext__app cp = app.Control_Panel id = panelClass.id if 0: # Enable to clean up the control panel. try: del cp._objects except: pass cp.id # Unghostify. if hasattr(cp, id): return cp._getOb(id) if cp.__dict__.has_key('_objects'): # _objects has been overridden. We have to persist. existing = getattr(aq_base(cp), id, None) if existing is None or existing.__class__ != panelClass: cp._setObject(id, panelClass()) else: # Don't persist what we don't have to. objects = ApplicationManager._objects objects = filter(lambda o, id=id: o['id'] != id, objects) ApplicationManager._objects = objects + ( {'id':id, 'meta_type':panelClass.meta_type},) try: delattr(cp, id) except: pass setattr(ApplicationManager, id, panelClass()) return cp._getOb(id) Where panelClass is my Product's class. When the ping script imports Zope, the product is installed. If it's installed as a "regular" product, using context.registerClass, all is OK. If it is installed using the above function, I get ZODB transaction notes of the form "Installed product CallProfiler by Zope 2002-04-08 11:14:10 PM". It's all a bit black magic for me though. I honestly don't know why half of the function above actually exists. I'm going to keep looking though, but if someone could help, I'd be extremely grateful. Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions
On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones <[EMAIL PROTECTED]> wrote: >I'm investigating why our product ZODB is growing when it shouldn't. I've >found that the undo log has entries in it every five minutes saying >"Installed product CallProfiler by Zope". More infrequently (not regularly) >we also have "Installed product ZOracleDA by Zope". Was zope restarting (or being restarted) every five minutes? Toby Dickenson [EMAIL PROTECTED] ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Call Profiler strangeness - lots of ZODB transactions
I'm investigating why our product ZODB is growing when it shouldn't. I've found that the undo log has entries in it every five minutes saying "Installed product CallProfiler by Zope". More infrequently (not regularly) we also have "Installed product ZOracleDA by Zope". The Call Profiler was designed specifically not to hit the ZODB - it's in-memory only with. Our setup consists of seven ZEO clients talking to the one ZEO server - the one that's getting all these hits in the ZODB. Does anyone else see this happening? Anyone have any ideas why we're seeing this behaviour? Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Mon, 4 Feb 2002 21:42, seb bacon wrote: > On Sun, 2002-02-03 at 21:58, Richard Jones wrote: > > On Fri, 1 Feb 2002 21:53, seb bacon wrote: > > > I wrote: > > > > I've just announced our Call Profiler product on zope.org and the > > > > zope announce list (waiting for people in different timezones to > > > > authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ > > > > for the impatient) > > > > > > This is absolutely excellent, well done and thanks :) I added various > > > FSObjects to the profiled modules list and have been looking at my CMF > > > systems - it's really instructive. > > > > Please send me the config entries for them. Also, if anyone has used ZPT > > with it, let me know if my guess of the config for them is correct :) > > in profileable_modules: > > 'FSPythonScript': Profileable('Products.CMFCore.FSPythonScript', > 'FSPythonScript', '__call__'), > 'FSDTMLMethod': Profileable('Products.CMFCore.FSDTMLMethod', > 'FSDTMLMethod', '__call__'), > 'FSPageTemplate': Profileable('Products.CMFCore.FSPageTemplate', > 'FSPageTemplate', '__call__'), Thanks! > > > - the trace for a complete template could be slightly clearer. The > > > colour coding for the start and end of a module could have a clear > > > legend, for example. > > > > I'll put a legend in. Is the colour coding what you meant by "clearer" or > > is there something else? > > Yes, that's what I meant. OK, it's done. Will be in the next release (probably today :) > Actually, another thing just occurred to me: > the meaning of the "..." ellipsis is not clear - I think you need a > different symbol to emphasise the hierarchy (perhaps a vertical bar?) OK, I'll look into it. The '...' is definitely a placeholder :) Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Sun, 2002-02-03 at 21:58, Richard Jones wrote: > On Fri, 1 Feb 2002 21:53, seb bacon wrote: > > I wrote: > > > I've just announced our Call Profiler product on zope.org and the zope > > > announce list (waiting for people in different timezones to authorise > > > them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the > > > impatient) > > > > This is absolutely excellent, well done and thanks :) I added various > > FSObjects to the profiled modules list and have been looking at my CMF > > systems - it's really instructive. > > Please send me the config entries for them. Also, if anyone has used ZPT with > it, let me know if my guess of the config for them is correct :) in profileable_modules: 'FSPythonScript': Profileable('Products.CMFCore.FSPythonScript', 'FSPythonScript', '__call__'), 'FSDTMLMethod': Profileable('Products.CMFCore.FSDTMLMethod', 'FSDTMLMethod', '__call__'), 'FSPageTemplate': Profileable('Products.CMFCore.FSPageTemplate', 'FSPageTemplate', '__call__'), > > You could do this check only if zope is being run in debug mode; then > > there would be zero perormance hit in production sites. > > No, there's still the performance check for "am I in debug mode?" which is > equivalent to "am I in profiling mode?". Sure, it's small, but it's there. > > - the trace for a complete template could be slightly clearer. The > > colour coding for the start and end of a module could have a clear > > legend, for example. > > I'll put a legend in. Is the colour coding what you meant by "clearer" or is > there something else? Yes, that's what I meant. Actually, another thing just occurred to me: the meaning of the "..." ellipsis is not clear - I think you need a different symbol to emphasise the hierarchy (perhaps a vertical bar?) Cheers, seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Sat, 2 Feb 2002 02:49, Zopista wrote: > Doesnt seem to like being refreshed and some event items arent in the time > dict, I noticed these were items that I had got in a ram cache. So i just > stuck an if at line 496. Other than that its a great product. Most > important tool since refresh if you ask me... Yeah, as the README states - the profiling data is memory-only - I'm going to look into a post-publish hook that stuffs the result from a single run into the ZODB. Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Fri, 1 Feb 2002 21:53, seb bacon wrote: > I wrote: > > I've just announced our Call Profiler product on zope.org and the zope > > announce list (waiting for people in different timezones to authorise > > them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the > > impatient) > > This is absolutely excellent, well done and thanks :) I added various > FSObjects to the profiled modules list and have been looking at my CMF > systems - it's really instructive. Please send me the config entries for them. Also, if anyone has used ZPT with it, let me know if my guess of the config for them is correct :) > > On one hand, the performance hit when dynamically patching the methods is > > zero when the product is not "active", but it does mean diddling with > > methods that really probably should be left alone. On the other hand, > > having changes to the core code to test for profiling being enabled > > introduces a small performance hit even when profiling is not activated. > > You could do this check only if zope is being run in debug mode; then > there would be zero perormance hit in production sites. No, there's still the performance check for "am I in debug mode?" which is equivalent to "am I in profiling mode?". Sure, it's small, but it's there. > > Any comments? > > I think the reporting could do with some usability improvements. It's > good now, but takes a while to navigate round. Yep, I know :) > - All the reports could benefit from being sortable by column. In the works already :) > - the trace for a complete template could be slightly clearer. The > colour coding for the start and end of a module could have a clear > legend, for example. I'll put a legend in. Is the colour coding what you meant by "clearer" or is there something else? > - there needs to be some solution for really long URLs, which require > lots of horizontal scrolling - just displaying the final parts of the > path, with the rest of the url hidden somehow? Yep, that's a good idea. > I can't think of exactly what else I would do to improve it right now I've already got some more code in place that I'm testing right now which does aggregation of matching result profiles. That is, if a request results in the same call profile, the timings are aggregated into min/average/max. Really neat :) Thanks for the feedback! Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
Doesnt seem to like being refreshed and some event items arent in the time dict, I noticed these were items that I had got in a ram cache. So i just stuck an if at line 496. Other than that its a great product. Most important tool since refresh if you ask me... - Original Message - From: "Richard Jones" <[EMAIL PROTECTED]> To: "zope-dev" <[EMAIL PROTECTED]> Sent: Thursday, January 31, 2002 5:13 PM Subject: [Zope-dev] Call Profiler > I've just announced our Call Profiler product on zope.org and the zope > announce list (waiting for people in different timezones to authorise them :) > ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) > > In a nutshell, it patches the rendering guts of DTMLMethod, ZSQL, etc. and > the ZPublisher so that we can get timing marks for requests as the calls are > made. The patching is quite simple (once I got the hang of it ;) and > reversible, and means that there's no changes required to the Zope core to > enable the testing (which our initial versions did). > > It does pose a question though: would it be better to have support for this > stuff in the core, or is it OK for a product like this to dynamically patch > the call methods when it needs to? > > On one hand, the performance hit when dynamically patching the methods is > zero when the product is not "active", but it does mean diddling with methods > that really probably should be left alone. On the other hand, having changes > to the core code to test for profiling being enabled introduces a small > performance hit even when profiling is not activated. > > Any comments? > > > Richard > > ___ > Zope-Dev maillist - [EMAIL PROTECTED] > http://lists.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://lists.zope.org/mailman/listinfo/zope-announce > http://lists.zope.org/mailman/listinfo/zope ) > ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On 1/2/02 1:13 am, "Richard Jones" <[EMAIL PROTECTED]> wrote: > I've just announced our Call Profiler product on zope.org and the zope > announce list (waiting for people in different timezones to authorise them :) > ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) > [snip] > Any comments? > I've been using it for *5* minutes and have found a method that was taking 95%! of one of our most often used scripts and another python script that was effectively stalling the servergobsmacked. Huge kudos and thanks to Richard!! :) Tone. -- Dr Tony McDonald, Assistant Director, FMCC, http://www.fmcc.org.uk/ The Medical School, Newcastle University Tel: +44 191 243 6140 A Zope list for UK HE/FE http://www.fmcc.org.uk/mailman/listinfo/zope ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
> I've just announced our Call Profiler product on zope.org and the zope > announce list (waiting for people in different timezones to authorise them :) > ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) This is absolutely excellent, well done and thanks :) I added various FSObjects to the profiled modules list and have been looking at my CMF systems - it's really instructive. > It does pose a question though: would it be better to have support for this > stuff in the core, or is it OK for a product like this to dynamically patch > the call methods when it needs to? I see nothing wrong with it being a monkeypatch product. On the other hand, it is a very useful tool, and as such could benefit from being in the core simply so that it always gets maintained with the core. > On one hand, the performance hit when dynamically patching the methods is > zero when the product is not "active", but it does mean diddling with methods > that really probably should be left alone. On the other hand, having changes > to the core code to test for profiling being enabled introduces a small > performance hit even when profiling is not activated. You could do this check only if zope is being run in debug mode; then there would be zero perormance hit in production sites. > > Any comments? > I think the reporting could do with some usability improvements. It's good now, but takes a while to navigate round. - All the reports could benefit from being sortable by column. - the trace for a complete template could be slightly clearer. The colour coding for the start and end of a module could have a clear legend, for example. - there needs to be some solution for really long URLs, which require lots of horizontal scrolling - just displaying the final parts of the path, with the rest of the url hidden somehow? I can't think of exactly what else I would do to improve it right now, though. And that's the kind of change I never get round to bothering with, personally... Sorry these aren't really constructive ideas, but you did ask for feedback :-) seb ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
This is really cool. The implementation looks pretty reasonable too! Definitely useful for folks doing development and wondering why their scripts take > 2 secs to run! ;-) I think this will be in many folks' toolbags soon. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
On Fri, 1 Feb 2002 14:10, Anthony Baxter wrote: > >>> Richard Jones wrote > > > > I've just announced our Call Profiler product on zope.org and the zope > > announce list (waiting for people in different timezones to authorise > > them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the > > impatient) > > ... Richard had made my initial vile hack actually work in a way that > didn't cause bleeding from the brain) No comment ;) > Unfortunately, the existing python profiling stuff is at too low a > level to be useful - this stuff really fills a need. Although being able to tie this into the python profiler would _rock_ in terms of profiling websites that use a mix of DTML/ZPT and Python products. Even it it's just to get the timing marks where the higher level stuff like DTML/ZPT calls out to the python code (ie. getting timing for the python calls at the highest level) > Plus, if it's integrated into Zope, Richard doesn't have to spend work > time on maintaining it :) But I do have time for screenshots - there's some up on the product page now for those who have a short curiosity span :) Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Call Profiler
>>> Richard Jones wrote > I've just announced our Call Profiler product on zope.org and the zope > announce list (waiting for people in different timezones to authorise them :) > ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) >From my point of view, this is a critical piece for Zope - right now, trying to work out what's going on in a Zope page render is somewhat painful. We found once we'd turned this onto our production code (and Richard had made my initial vile hack actually work in a way that didn't cause bleeding from the brain) we were able to very very quickly spot a whole lot of stuff that was Just Not Doing The Right Thing. Unfortunately, the existing python profiling stuff is at too low a level to be useful - this stuff really fills a need. Plus, if it's integrated into Zope, Richard doesn't have to spend work time on maintaining it :) Anthony ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
[Zope-dev] Call Profiler
I've just announced our Call Profiler product on zope.org and the zope announce list (waiting for people in different timezones to authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/ for the impatient) In a nutshell, it patches the rendering guts of DTMLMethod, ZSQL, etc. and the ZPublisher so that we can get timing marks for requests as the calls are made. The patching is quite simple (once I got the hang of it ;) and reversible, and means that there's no changes required to the Zope core to enable the testing (which our initial versions did). It does pose a question though: would it be better to have support for this stuff in the core, or is it OK for a product like this to dynamically patch the call methods when it needs to? On one hand, the performance hit when dynamically patching the methods is zero when the product is not "active", but it does mean diddling with methods that really probably should be left alone. On the other hand, having changes to the core code to test for profiling being enabled introduces a small performance hit even when profiling is not activated. Any comments? Richard ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )