Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions

2002-04-08 Thread Richard Jones

On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote:
 On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones

 [EMAIL PROTECTED] wrote:
 I'm investigating why our product ZODB is growing when it shouldn't. I've
 found that the undo log has entries in it every five minutes saying
 Installed product CallProfiler by Zope. More infrequently (not
  regularly) we also have Installed product ZOracleDA by Zope.

 Was zope restarting (or being restarted) every five minutes?

No, but it is being pinged. By that, I mean that we have a script that tries 
to access the ZEO server every five minutes to make sure it's all OK. The 
script does the following::

  import sys, os
  sys.path.append('/app/zope/zopecontrol_code/')
  sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python')
  os.environ['ZEOPORT'] = '8076'
  os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com'
  import Zope
  # now do stuff with the Zope.app() (we don't actually have to do anything
  # to trigger the ZODB note generation - the import is enough)

Now, the problem lies in the way that the CallProfiler gets itself into the 
Control Panel. This code was copied from the LeakFinder product, so I assume 
it'll have the same problem. The code is::

 def installControlPanel(context, panelClass):
from App.ApplicationManager import ApplicationManager
from Acquisition import aq_base
app = context._ProductContext__app
cp = app.Control_Panel
id = panelClass.id
if 0: # Enable to clean up the control panel.
try: del cp._objects
except: pass
cp.id # Unghostify.
if hasattr(cp, id):
return cp._getOb(id)
if cp.__dict__.has_key('_objects'):
# _objects has been overridden.  We have to persist.
existing = getattr(aq_base(cp), id, None)
if existing is None or existing.__class__ != panelClass:
cp._setObject(id, panelClass())
else:
# Don't persist what we don't have to.
objects = ApplicationManager._objects
objects = filter(lambda o, id=id: o['id'] != id, objects)
ApplicationManager._objects = objects + (
{'id':id, 'meta_type':panelClass.meta_type},)
try: delattr(cp, id)
except: pass
setattr(ApplicationManager, id, panelClass())
return cp._getOb(id)

Where panelClass is my Product's class.

When the ping script imports Zope, the product is installed. If it's 
installed as a regular product, using context.registerClass, all is OK. If 
it is installed using the above function, I get ZODB transaction notes of the 
form Installed product CallProfiler by Zope   2002-04-08 11:14:10 PM.

It's all a bit black magic for me though. I honestly don't know why half of 
the function above actually exists. I'm going to keep looking though, but if 
someone could help, I'd be extremely grateful.


Richard


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions

2002-04-08 Thread Richard Jones

On Tue, 9 Apr 2002 09:27, Richard Jones wrote:
 On Tue, 26 Mar 2002 21:29, Toby Dickenson wrote:
  On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones
 
  [EMAIL PROTECTED] wrote:
  I'm investigating why our product ZODB is growing when it shouldn't.
   I've found that the undo log has entries in it every five minutes
   saying Installed product CallProfiler by Zope. More infrequently (not
   regularly) we also have Installed product ZOracleDA by Zope.
 
  Was zope restarting (or being restarted) every five minutes?

 No, but it is being pinged. By that, I mean that we have a script that
 tries to access the ZEO server every five minutes to make sure it's all OK.
 The script does the following::

   import sys, os
   sys.path.append('/app/zope/zopecontrol_code/')
   sys.path.insert(0, '/app/zope/zopecontrol_code/lib/python')
   os.environ['ZEOPORT'] = '8076'
   os.environ['ZEOHOST'] = 'devhost1.off.ekorp.com'
   import Zope
   # now do stuff with the Zope.app() (we don't actually have to do anything
   # to trigger the ZODB note generation - the import is enough)

Setting the environment var ZEO_CLIENT stops the message - but I guess I'd 
like to know why this product is causing the message when none of the others 
are. It's all due to the code in installControlPanel, and I'd rather it not 
be dodgy :)


Richard


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler strangeness - lots of ZODB transactions

2002-03-26 Thread Toby Dickenson

On Tue, 26 Mar 2002 15:30:56 +1100, Richard Jones
[EMAIL PROTECTED] wrote:

I'm investigating why our product ZODB is growing when it shouldn't. I've 
found that the undo log has entries in it every five minutes saying 
Installed product CallProfiler by Zope. More infrequently (not regularly) 
we also have Installed product ZOracleDA by Zope.

Was zope restarting (or being restarted) every five minutes?



Toby Dickenson
[EMAIL PROTECTED]

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-02-04 Thread Richard Jones

On Mon, 4 Feb 2002 21:42, seb bacon wrote:
 On Sun, 2002-02-03 at 21:58, Richard Jones wrote:
  On Fri, 1 Feb 2002 21:53, seb bacon wrote:
   I wrote:
I've just announced our Call Profiler product on zope.org and the
zope announce list (waiting for people in different timezones to
authorise them :) ( http://www.zope.org/Members/richard/CallProfiler/
 for the impatient)
  
   This is absolutely excellent, well done and thanks :)  I added various
   FSObjects to the profiled modules list and have been looking at my CMF
   systems - it's really instructive.
 
  Please send me the config entries for them. Also, if anyone has used ZPT
  with it, let me know if my guess of the config for them is correct :)

 in profileable_modules:

 'FSPythonScript': Profileable('Products.CMFCore.FSPythonScript',
 'FSPythonScript', '__call__'),
 'FSDTMLMethod': Profileable('Products.CMFCore.FSDTMLMethod',
 'FSDTMLMethod', '__call__'),
 'FSPageTemplate': Profileable('Products.CMFCore.FSPageTemplate',
 'FSPageTemplate', '__call__'),

Thanks!


   - the trace for a complete template could be slightly clearer.  The
   colour coding for the start and end of a module could have a clear
   legend, for example.
 
  I'll put a legend in. Is the colour coding what you meant by clearer or
  is there something else?

 Yes, that's what I meant.

OK, it's done. Will be in the next release (probably today :)


  Actually, another thing just occurred to me:
 the meaning of the ... ellipsis is not clear - I think you need a
 different symbol to emphasise the hierarchy (perhaps a vertical bar?)

OK, I'll look into it. The '...' is definitely a placeholder :)


Richard

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-02-03 Thread Richard Jones

On Fri, 1 Feb 2002 21:53, seb bacon wrote:
 I wrote:
  I've just announced our Call Profiler product on zope.org and the zope
  announce list (waiting for people in different timezones to authorise
  them :) ( http://www.zope.org/Members/richard/CallProfiler/  for the
  impatient)

 This is absolutely excellent, well done and thanks :)  I added various
 FSObjects to the profiled modules list and have been looking at my CMF
 systems - it's really instructive.

Please send me the config entries for them. Also, if anyone has used ZPT with 
it, let me know if my guess of the config for them is correct :)


  On one hand, the performance hit when dynamically patching the methods is
  zero when the product is not active, but it does mean diddling with
  methods that really probably should be left alone. On the other hand,
  having changes to the core code to test for profiling being enabled
  introduces a small performance hit even when profiling is not activated.

 You could do this check only if zope is being run in debug mode; then
 there would be zero perormance hit in production sites.

No, there's still the performance check for am I in debug mode? which is 
equivalent to am I in profiling mode?. Sure, it's small, but it's there.


  Any comments?

 I think the reporting could do with some usability improvements.  It's
 good now, but takes a while to navigate round.

Yep, I know :)


 - All the reports could benefit from being sortable by column.

In the works already :)


 - the trace for a complete template could be slightly clearer.  The
 colour coding for the start and end of a module could have a clear
 legend, for example.

I'll put a legend in. Is the colour coding what you meant by clearer or is 
there something else? 


 - there needs to be some solution for really long URLs, which require
 lots of horizontal scrolling - just displaying the final parts of the
 path, with the rest of the url hidden somehow?

Yep, that's a good idea.


 I can't think of exactly what else I would do to improve it right now

I've already got some more code in place that I'm testing right now which 
does aggregation of matching result profiles. That is, if a request results 
in the same call profile, the timings are aggregated into min/average/max. 
Really neat :)

Thanks for the feedback!


Richard

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-02-01 Thread seb bacon

 I've just announced our Call Profiler product on zope.org and the zope 
 announce list (waiting for people in different timezones to authorise them :)
   ( http://www.zope.org/Members/richard/CallProfiler/  for the impatient)

This is absolutely excellent, well done and thanks :)  I added various FSObjects to 
the profiled modules list and have been looking at my CMF systems - it's really 
instructive.

 It does pose a question though: would it be better to have support for this 
 stuff in the core, or is it OK for a product like this to dynamically patch 
 the call methods when it needs to?

I see nothing wrong with it being a monkeypatch product.  On the other
hand, it is a very useful tool, and as such could benefit from being in
the core simply so that it always gets maintained with the core.

 On one hand, the performance hit when dynamically patching the methods is 
 zero when the product is not active, but it does mean diddling with methods 
 that really probably should be left alone. On the other hand, having changes 
 to the core code to test for profiling being enabled introduces a small 
 performance hit even when profiling is not activated.

You could do this check only if zope is being run in debug mode; then
there would be zero perormance hit in production sites.

 
 Any comments?
 

I think the reporting could do with some usability improvements.  It's
good now, but takes a while to navigate round.  

- All the reports could benefit from being sortable by column.  
- the trace for a complete template could be slightly clearer.  The
colour coding for the start and end of a module could have a clear
legend, for example.
- there needs to be some solution for really long URLs, which require
lots of horizontal scrolling - just displaying the final parts of the
path, with the rest of the url hidden somehow?

I can't think of exactly what else I would do to improve it right now,
though.  And that's the kind of change I never get round to bothering
with, personally... Sorry these aren't really constructive ideas, but
you did ask for feedback :-)

seb


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-02-01 Thread Tony McDonald

On 1/2/02 1:13 am, Richard Jones [EMAIL PROTECTED] wrote:

 I've just announced our Call Profiler product on zope.org and the zope
 announce list (waiting for people in different timezones to authorise them :)
 ( http://www.zope.org/Members/richard/CallProfiler/  for the impatient)
 

[snip]

 Any comments?
 

I've been using it for *5* minutes and have found a method that was taking
95%! of one of our most often used scripts and another python script that
was effectively stalling the servergobsmacked.

Huge kudos and thanks to Richard!! :)

Tone.
-- 
Dr Tony McDonald,  Assistant Director, FMCC, http://www.fmcc.org.uk/
The Medical School, Newcastle University Tel: +44 191 243 6140
A Zope list for UK HE/FE  http://www.fmcc.org.uk/mailman/listinfo/zope


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-02-01 Thread Zopista

Doesnt seem to like being refreshed and some event items arent in the time
dict, I noticed these were items that I had got in a ram cache. So i just
stuck an if at line 496. Other than that its a great product. Most important
tool since refresh if you ask me...

- Original Message -
From: Richard Jones [EMAIL PROTECTED]
To: zope-dev [EMAIL PROTECTED]
Sent: Thursday, January 31, 2002 5:13 PM
Subject: [Zope-dev] Call Profiler


 I've just announced our Call Profiler product on zope.org and the zope
 announce list (waiting for people in different timezones to authorise them
:)
   ( http://www.zope.org/Members/richard/CallProfiler/  for the impatient)

 In a nutshell, it patches the rendering guts of DTMLMethod, ZSQL, etc. and
 the ZPublisher so that we can get timing marks for requests as the calls
are
 made. The patching is quite simple (once I got the hang of it ;) and
 reversible, and means that there's no changes required to the Zope core to
 enable the testing (which our initial versions did).

 It does pose a question though: would it be better to have support for
this
 stuff in the core, or is it OK for a product like this to dynamically
patch
 the call methods when it needs to?

 On one hand, the performance hit when dynamically patching the methods is
 zero when the product is not active, but it does mean diddling with
methods
 that really probably should be left alone. On the other hand, having
changes
 to the core code to test for profiling being enabled introduces a small
 performance hit even when profiling is not activated.

 Any comments?


 Richard

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )




___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-01-31 Thread Anthony Baxter

 Richard Jones wrote
 I've just announced our Call Profiler product on zope.org and the zope 
 announce list (waiting for people in different timezones to authorise them :)
   ( http://www.zope.org/Members/richard/CallProfiler/  for the impatient)

From my point of view, this is a critical piece for Zope - right now, 
trying to work out what's going on in a Zope page render is somewhat
painful. We found once we'd turned this onto our production code (and
Richard had made my initial vile hack actually work in a way that didn't
cause bleeding from the brain) we were able to very very quickly spot
a whole lot of stuff that was Just Not Doing The Right Thing. 

Unfortunately, the existing python profiling stuff is at too low a 
level to be useful - this stuff really fills a need. 

Plus, if it's integrated into Zope, Richard doesn't have to spend work 
time on maintaining it :)

Anthony

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-01-31 Thread Richard Jones

On Fri, 1 Feb 2002 14:10, Anthony Baxter wrote:
  Richard Jones wrote
 
  I've just announced our Call Profiler product on zope.org and the zope
  announce list (waiting for people in different timezones to authorise
  them :) ( http://www.zope.org/Members/richard/CallProfiler/  for the
  impatient)

 ... Richard had made my initial vile hack actually work in a way that
 didn't cause bleeding from the brain) 

No comment ;)


 Unfortunately, the existing python profiling stuff is at too low a
 level to be useful - this stuff really fills a need.

Although being able to tie this into the python profiler would _rock_ in 
terms of profiling websites that use a mix of DTML/ZPT and Python products. 
Even it it's just to get the timing marks where the higher level stuff like 
DTML/ZPT calls out to the python code (ie. getting timing for the python 
calls at the highest level)


 Plus, if it's integrated into Zope, Richard doesn't have to spend work
 time on maintaining it :)

But I do have time for screenshots - there's some up on the product page now 
for those who have a short curiosity span :)


Richard

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Call Profiler

2002-01-31 Thread Chris McDonough

This is really cool.  The implementation looks pretty reasonable too! 
Definitely useful for folks doing development and wondering why their 
scripts take  2 secs to run!  ;-)  I think this will be in many folks' 
toolbags soon.

- C


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )