Re: [Zope-dev] Zope Server hanging :-(

2001-06-06 Thread Erik Enge

On Wed, 6 Jun 2001 [EMAIL PROTECTED] wrote:

 ... at least then we know what the exception is.
: 
 Again - try that code in an interactive interpreter if you really want to
 find out what's going on...

Yeah, thanks, that would work as a workaround, but isn't this buggish
behaviour?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-06 Thread Richard Jones

On Wednesday 06 June 2001 18:03, Erik Enge wrote:
 On Wed, 6 Jun 2001 [EMAIL PROTECTED] wrote:
  ... at least then we know what the exception is.
 
  Again - try that code in an interactive interpreter if you really want to
  find out what's going on...

 Yeah, thanks, that would work as a workaround, but isn't this buggish
 behaviour?

Absolutely - along with dozens of other places that Zope squashes exceptions. 
Anthony Baxter actually wrote a script that finds instances of bare except: 
clauses in the Zope source...

 http://www.zope.org/Members/anthony/BarewordExcepts

Feel free to find the bad except: and submit a patch...


Richard

-- 
Richard Jones
[EMAIL PROTECTED]
Senior Software Developer, Bizar Software (www.bizarsoftware.com.au)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-06 Thread Erik Enge

On Wed, 6 Jun 2001, Richard Jones wrote:

  http://www.zope.org/Members/anthony/BarewordExcepts
 
 Feel free to find the bad except: and submit a patch...

Ugh.  There are tons of them...  I'll see what I have time for.


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-05 Thread Erik Enge

On Sun, 29 Apr 2001, Erik Enge wrote:

 I'll try to do an hour or so of analysing this tomorrow, and I'll get back
 to you.  :-)

Well, now we all know what vikings believe an hour or so mean, don't we?

I figured it out, I think.  Let's say I have these two methods:

 def a():
b()

 def b():
a()

If I call a(), then Zope dies and restarts without giving me any error
at all.  Anyone got a clue?  Makes debugging a bit tedious sometimes.
I added a couple of print's in the methods and it seem to call them both
many times before dying.

I seem to remeber some ExessiveRecursion or some such error that popped up
on me a year or two ago, has that left for happier hunting fields?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-05 Thread richard

Erik Enge wrote:
 On Sun, 29 Apr 2001, Erik Enge wrote:
 I figured it out, I think.  Let's say I have these two methods:
 
  def a():
 b()
 
  def b():
 a()
 
 If I call a(), then Zope dies and restarts without giving me any error
 at all.  Anyone got a clue?

There are many places in Zope where errors such as this (RuntimeError - try
it in an interactive interpreter) are silently swallowed by too-broad
except: clauses (except in this case, the exception is eaten, but the
interpreter process exits anyway...).

When we find that our code is dying without notice, we usually try to
narrow down the area of code that could be to blame, and then wrap it in
our own try/except like:

  try:
do possibly bad stuff
  except:
import traceback
traceback.print_exc()
raise

... at least then we know what the exception is.


 I seem to remeber some ExessiveRecursion or some such error that popped up
 on me a year or two ago, has that left for happier hunting fields?

Again - try that code in an interactive interpreter if you really want to
find out what's going on...


Richard

-- 
Richard Jones
[EMAIL PROTECTED]
Senior Software Developer, Bizar Software (www.bizarsoftware.com.au)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-05 Thread Dieter Maurer

  Erik Enge wrote:
   On Sun, 29 Apr 2001, Erik Enge wrote:
   I figured it out, I think.  Let's say I have these two methods:
   
def a():
   b()
   
def b():
   a()
   
   If I call a(), then Zope dies and restarts without giving me any error
   at all.  Anyone got a clue?
This is an infinite recursion.

I once had such a situation.

   Usually Python protects itself against such recursions by limiting
   the depth of its runtime stack. It raises a RuntimeError: runtime
   stack limit exceeded when its stack overflows.

   But in my case, the thread's runtime stack (maintained
   by the C runtime not Python) was more limited
   than the Python stack limit. When the thread's stack overflew,
   the process was killed by Solaris.
   Python did not have any chance to raise an exception
   as the death was immediate.


We may need to ask the Python maintainers to increase the
thread stack size or to more severely restrict the
Python runtime stack.


Dieter



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-06-05 Thread richard

Dieter Maurer wrote:
 
   Erik Enge wrote:
On Sun, 29 Apr 2001, Erik Enge wrote:
I figured it out, I think.  Let's say I have these two methods:
   
 def a():
b()
   
 def b():
a()
   
If I call a(), then Zope dies and restarts without giving me any error
at all.  Anyone got a clue?
 This is an infinite recursion.
 
 I once had such a situation.
 
Usually Python protects itself against such recursions by limiting
the depth of its runtime stack. It raises a RuntimeError: runtime
stack limit exceeded when its stack overflows.
 
But in my case, the thread's runtime stack (maintained
by the C runtime not Python) was more limited
than the Python stack limit. When the thread's stack overflew,
the process was killed by Solaris.
Python did not have any chance to raise an exception
as the death was immediate.
 
 We may need to ask the Python maintainers to increase the
 thread stack size or to more severely restrict the
 Python runtime stack.

This can be done manually in your site.py - sys.setrecursionlimit()


 Richard

-- 
Richard Jones
[EMAIL PROTECTED]
Senior Software Developer, Bizar Software (www.bizarsoftware.com.au)

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging: DataPoint

2001-05-03 Thread Chris Withers

Hmmm... saw these in the stupid log, just before the server crashed:

--
2001-05-03T10:09:34 INFO(0) Z2 CONFLICT Competing writes at,
/VirtualHostBase/http/server.nipltd.com:80/VirtualHostRoot/folder/object
Traceback (innermost last):
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
171, in publish
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/mapply.py, line 160,
in mapply
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
112, in call_object
snip lots of Python Scripts calling each other and DTML methods
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Connection.py, line 524,
in setstate
ConflictError: ('\\000\\000\\000\\000\\000\\020\\036\\304', 'extension class
Products.AProduct.Utility.AClass at 84ea878')

--
2001-05-03T10:09:42 INFO(0) Z2 CONFLICT Competing writes at,
/VirtualHostBase/http/server.nipltd.com:80/VirtualHostRoot/folder/object
Traceback (innermost last):
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
175, in publish
  File /usr/local/zope/Zope-2.3.2-src/lib/python/Zope/__init__.py, line 235, in
commit
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Transaction.py, line 300,
in commit
  File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Connection.py, line 281,
in commit
ConflictError: '\000\000\000\000\000\020\037\245'

Could this bring it down? Should it? How can I track it down further?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging: DataPoint

2001-05-03 Thread Chris Withers

Good morning, I presume? :-)

Chris McDonough wrote:
 
 This probably isn't meaningful... unless it is.  ;-) 

Huh?

 Conflict errors
 are normal... I wish they wouldn't be so big and have an UPPERCASE
 word in their description.

OK, but I did notice them just before the server died. Maybe now would be a good
time to run the analyser script.
Unfortunately, I'd probably be expected to 'clean' the URLs of sensitive
information.

Hurm, d'ya think it'd be worth it?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging: DataPoint

2001-05-03 Thread Chris McDonough

Chris Withers wrote:
 
 Good morning, I presume? :-)

Present!

  This probably isn't meaningful... unless it is.  ;-)
 
 Huh?

This is a Jim Fultonism.  In English, it means this is probably not
meaningful.

  Conflict errors
  are normal... I wish they wouldn't be so big and have an UPPERCASE
  word in their description.
 
 OK, but I did notice them just before the server died. Maybe now would be a good
 time to run the analyser script.
 Unfortunately, I'd probably be expected to 'clean' the URLs of sensitive
 information.

Well the analyzer script only shows the first 20 or so chars of the URL
by default unless you pass it the --verbose argument, so this might
serve to do some cleaning.  Also, getting rid of the URL altogether is
something you can do by using Unix cut or on windows putting the
results of the analyzer script into Excel and deleting the column with
the URL in it.
 
 Hurm, d'ya think it'd be worth it?

It's up to you, but I think so.

- C

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



RE: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Tim McLaughlin

Sorry, Chris M, didn't mean to get you stuck in the documentation torture
chamber ;-)

Cheers,
Tim

-Original Message-
From: Michel Pelletier [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 02, 2001 11:58 PM
To: Chris McDonough
Cc: Tim McLaughlin; Chris Withers; [EMAIL PROTECTED]
Subject: Re: [Zope-dev] Zope Server hanging :-(


On Wed, 2 May 2001, Chris McDonough wrote:

 The docs (available via the --help switch) go into some of this detail,
but
 I agree that a narrative explaining how to approach it from a functional
 perspective would be a good thing.

This should go in the debuggin and testing chapter of the dev guide.
Wanna take a stab?

-Michel

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Chris Withers

Chris McDonough wrote:
 
 You can also use cumulative reporting to sort by hangs:
 
   requestprofiler.py your.log.file --cumulative --sort=hangs

python requestprofiler.py my.log --cumulative --sort=hangs  out2.txt

resulted in:

Traceback (innermost last):
  File requestprofiler.py, line 586, in ?
analyze(open(sys.argv[1]), top, sortf, start, end, mode)
  File requestprofiler.py, line 346, in analyze
write(dict, top)
  File requestprofiler.py, line 360, in write
print str(stat)[:78]
  File requestprofiler.py, line 226, in __str__
body = (
  File requestprofiler.py, line 273, in median
else: return (all[i] + all[i2]) / 2
TypeError: number coercion failed

Any ideas?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Chris McDonough

Please apply this patch, sorry!

*** requestprofiler.py 2001/04/26 02:51:49 1.3
--- requestprofiler.py 2001/05/03 15:12:13
***
*** 269,276 
  i2 = i + 1
  v1 = all[i]
  v2 = all[i2]
! if v1 == NA or v2 == NA: return I
! else: return (all[i] + all[i2]) / 2

  def total(self):
  t = 0
--- 269,276 
  i2 = i + 1
  v1 = all[i]
  v2 = all[i2]
! if type(v1) is type('') or type(v2) is type(''): return
I
! else: return (v1 + v2) / 2

  def total(self):
  t = 0

- Original Message -
From: Chris Withers [EMAIL PROTECTED]
To: Chris McDonough [EMAIL PROTECTED]
Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Thursday, May 03, 2001 10:35 AM
Subject: Re: [Zope-dev] Zope Server hanging :-(


 Chris McDonough wrote:
 
  You can also use cumulative reporting to sort by hangs:
 
requestprofiler.py your.log.file --cumulative --sort=hangs

 python requestprofiler.py my.log --cumulative --sort=hangs  out2.txt

 resulted in:

 Traceback (innermost last):
   File requestprofiler.py, line 586, in ?
 analyze(open(sys.argv[1]), top, sortf, start, end, mode)
   File requestprofiler.py, line 346, in analyze
 write(dict, top)
   File requestprofiler.py, line 360, in write
 print str(stat)[:78]
   File requestprofiler.py, line 226, in __str__
 body = (
   File requestprofiler.py, line 273, in median
 else: return (all[i] + all[i2]) / 2
 TypeError: number coercion failed

 Any ideas?

 cheers,

 Chris



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Michel Pelletier

On Thu, 3 May 2001, Chris McDonough wrote:

 It's no problem, especially as Michel had insomnia last night and did
 it.  ;-)

My best stuff comes out when I can't sleep.

Those of you who are interested in looking at it, check out the
ever-growing Zope Developer's Guide (shame on you if you haven't read
it!).

http://sourceforge.net/projects/zope-devel/

-Michel


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Michel Pelletier

On Thu, 3 May 2001, Michel Pelletier wrote:

 On Thu, 3 May 2001, Chris McDonough wrote:
 
  It's no problem, especially as Michel had insomnia last night and did
  it.  ;-)
 
 My best stuff comes out when I can't sleep.
 
 Those of you who are interested in looking at it, check out the
 ever-growing Zope Developer's Guide (shame on you if you haven't read
 it!).

Ok, well maybe not shame considering you have to check it out of CVS. ;0

-Michel


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-03 Thread Chris McDonough

Michel Pelletier wrote:
 
  Those of you who are interested in looking at it, check out the
  ever-growing Zope Developer's Guide (shame on you if you haven't read
  it!).
 
 Ok, well maybe not shame considering you have to check it out of CVS. ;0

Just for folks who aren't following closely, Michel means that you can't
see the most current new shiny development version without checking it
out of CVS, but you *can* read a rough draft as of last week online. 
It's available on Zope.org at http://www.zope.org/Documentation/ZDG .

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris Withers

Chris McDonough wrote:
 
 Absolutely!  When you've got some representative data, and you've
 successfully run requestprofiler against it in various ways, let me know.

Hmmm, not really sure what I should be looking for :-S

What ways should I look to run it and what should I do with the output?

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris McDonough

  Attached is a script that I just checked into the trunk to do analysis
of
  the file generated by the -M log.  It can help you figure out if there's
a
  pattern to the hangs (whether it happens on a particular method, whether
it
  happens at heavy load time, whether it happens at a particular time of
day,
  etc.)  Do you think you'd be willing to play around with it a little bit
to
  try to discover a pattern?

 Definitely, can you help me with 'reading the runes'?

Absolutely!  When you've got some representative data, and you've
successfully run requestprofiler against it in various ways, let me know.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris McDonough

One of the more important bits is the active count when using
the --detailed option to requestprofiler.  This tells you how many *other*
requests were unfinished at the end of a particular request.

So for instance, if you run requestprofiler like this:

  python requestprofiler.py log.file.name --detailed

And you see a profile line like (header added for reference):

startwin wout wend tot S   osize  code act url
2001-04-19T01:19:420   120  12 E   26024  2005
/Catalog/checkrep?P

This means that:

 - this request started at 2001-04-19 1:19:42 UTC (start)

 - zserver waited 0 secs for input from the client (win)

 - zserver waited 12 secs for output from the publisher (wend)

 - the total time from start to end for the request was 12 secs (tot)

 - the request reached the end (E) stage (S, one of E, A, I, B)

 - the output size in bytes was 26024 (osize)

 - the http code was 200 (code)

 - the number of pending requests at the end of this request was 5
(active)

 - the URL of the request (url)

Hangs are also interesting.

Use detailed reporting to sort by end stage:

  requestprofiler.py your.log.file --detailed --sort=endstage

Any requests that don't show a stage of E for their endstage were hung
requests.

You can also use cumulative reporting to sort by hangs:

  requestprofiler.py your.log.file --cumulative --sort=hangs

This will show the methods that hung most frequently.  Much of the time,
you can find problems this way by seeing methods that enter an infinte loop
or cause a segfault.

Total time is also interesting:

  requestprofiler.py your.log.file --detailed --sort=total

  Shows you individual calls to methods that took a long time.

  - AND -

  requestprofiler.py your.log.file --cumulative --sort=total

   Shows you where most of the request time is spent.

The goal is to indentify methods or requests that take a long time, don't
return, or are accessed frequently.  If you can match this data up with
specific problems you've experienced (possibly by way of times/dates), it
becomes easier to form postulations about what is happening to Zope under
load.  Then investigate those postulations by refactoring code or by doing
more detailed logging in the places where you think the problems are.

- C





 Hmmm, not really sure what I should be looking for :-S

 What ways should I look to run it and what should I do with the output?

 cheers,

 Chris



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris Withers

Chris McDonough wrote:
 
snip info

Thanks, that's just what I was looking for

 The goal is to indentify methods or requests that take a long time, don't
 return, or are accessed frequently.  

Yup

 If you can match this data up with
 specific problems you've experienced (possibly by way of times/dates),

well, I have tail -f of the detailed log constantly open on my desktop ;-)

 it
 becomes easier to form postulations about what is happening to Zope under
 load. 

To be honest, there really isn't any load on this server... maybe one or two
people using it :-S

 Then investigate those postulations by refactoring code or by doing
 more detailed logging in the places where you think the problems are.

Hmmm, I know what you mean but it doesn't sound like it's going to be fun :-S

well, thanks for the info, I'll do what I can...

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



RE: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Tim McLaughlin

Nice info.  This could be useful for many, many things.  you oughtta put
this in a how-to (if you haven't already).  thanks Chris.

I'll use it if we have any more issues.  Currently, as I told ChrisW, the
issues have stopped. But I'm still keeping my eye on it.

-Original Message-
From: Chris McDonough [mailto:[EMAIL PROTECTED]]
Sent: Wednesday, May 02, 2001 12:00 PM
To: Chris Withers
Cc: Tim McLaughlin; [EMAIL PROTECTED]
Subject: Re: [Zope-dev] Zope Server hanging :-(


One of the more important bits is the active count when using
the --detailed option to requestprofiler.  This tells you how many *other*
requests were unfinished at the end of a particular request.

So for instance, if you run requestprofiler like this:

  python requestprofiler.py log.file.name --detailed

And you see a profile line like (header added for reference):

startwin wout wend tot S   osize  code act url
2001-04-19T01:19:420   120  12 E   26024  2005
/Catalog/checkrep?P

This means that:

 - this request started at 2001-04-19 1:19:42 UTC (start)

 - zserver waited 0 secs for input from the client (win)

 - zserver waited 12 secs for output from the publisher (wend)

 - the total time from start to end for the request was 12 secs (tot)

 - the request reached the end (E) stage (S, one of E, A, I, B)

 - the output size in bytes was 26024 (osize)

 - the http code was 200 (code)

 - the number of pending requests at the end of this request was 5
(active)

 - the URL of the request (url)

Hangs are also interesting.

Use detailed reporting to sort by end stage:

  requestprofiler.py your.log.file --detailed --sort=endstage

Any requests that don't show a stage of E for their endstage were hung
requests.

You can also use cumulative reporting to sort by hangs:

  requestprofiler.py your.log.file --cumulative --sort=hangs

This will show the methods that hung most frequently.  Much of the time,
you can find problems this way by seeing methods that enter an infinte loop
or cause a segfault.

Total time is also interesting:

  requestprofiler.py your.log.file --detailed --sort=total

  Shows you individual calls to methods that took a long time.

  - AND -

  requestprofiler.py your.log.file --cumulative --sort=total

   Shows you where most of the request time is spent.

The goal is to indentify methods or requests that take a long time, don't
return, or are accessed frequently.  If you can match this data up with
specific problems you've experienced (possibly by way of times/dates), it
becomes easier to form postulations about what is happening to Zope under
load.  Then investigate those postulations by refactoring code or by doing
more detailed logging in the places where you think the problems are.

- C





 Hmmm, not really sure what I should be looking for :-S

 What ways should I look to run it and what should I do with the output?

 cheers,

 Chris


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris McDonough

The docs (available via the --help switch) go into some of this detail, but
I agree that a narrative explaining how to approach it from a functional
perspective would be a good thing.


- Original Message -
From: Tim McLaughlin [EMAIL PROTECTED]
To: 'Chris McDonough' [EMAIL PROTECTED]; Chris Withers
[EMAIL PROTECTED]
Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Wednesday, May 02, 2001 12:05 PM
Subject: RE: [Zope-dev] Zope Server hanging :-(


 Nice info.  This could be useful for many, many things.  you oughtta put
 this in a how-to (if you haven't already).  thanks Chris.

 I'll use it if we have any more issues.  Currently, as I told ChrisW, the
 issues have stopped. But I'm still keeping my eye on it.

 -Original Message-
 From: Chris McDonough [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, May 02, 2001 12:00 PM
 To: Chris Withers
 Cc: Tim McLaughlin; [EMAIL PROTECTED]
 Subject: Re: [Zope-dev] Zope Server hanging :-(


 One of the more important bits is the active count when using
 the --detailed option to requestprofiler.  This tells you how many *other*
 requests were unfinished at the end of a particular request.

 So for instance, if you run requestprofiler like this:

   python requestprofiler.py log.file.name --detailed

 And you see a profile line like (header added for reference):

 startwin wout wend tot S   osize  code act url
 2001-04-19T01:19:420   120  12 E   26024  2005
 /Catalog/checkrep?P

 This means that:

  - this request started at 2001-04-19 1:19:42 UTC (start)

  - zserver waited 0 secs for input from the client (win)

  - zserver waited 12 secs for output from the publisher (wend)

  - the total time from start to end for the request was 12 secs (tot)

  - the request reached the end (E) stage (S, one of E, A, I,
B)

  - the output size in bytes was 26024 (osize)

  - the http code was 200 (code)

  - the number of pending requests at the end of this request was 5
 (active)

  - the URL of the request (url)

 Hangs are also interesting.

 Use detailed reporting to sort by end stage:

   requestprofiler.py your.log.file --detailed --sort=endstage

 Any requests that don't show a stage of E for their endstage were hung
 requests.

 You can also use cumulative reporting to sort by hangs:

   requestprofiler.py your.log.file --cumulative --sort=hangs

 This will show the methods that hung most frequently.  Much of the time,
 you can find problems this way by seeing methods that enter an infinte
loop
 or cause a segfault.

 Total time is also interesting:

   requestprofiler.py your.log.file --detailed --sort=total

   Shows you individual calls to methods that took a long time.

   - AND -

   requestprofiler.py your.log.file --cumulative --sort=total

Shows you where most of the request time is spent.

 The goal is to indentify methods or requests that take a long time, don't
 return, or are accessed frequently.  If you can match this data up with
 specific problems you've experienced (possibly by way of times/dates), it
 becomes easier to form postulations about what is happening to Zope under
 load.  Then investigate those postulations by refactoring code or by doing
 more detailed logging in the places where you think the problems are.

 - C





  Hmmm, not really sure what I should be looking for :-S
 
  What ways should I look to run it and what should I do with the output?
 
  cheers,
 
  Chris
 



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Chris Muldrow

I'll vouch for the effectiveness of this method in tracking down some
problems. Chris' advice helped us track down some horrendous numbers around
a method calling DateTime. We've seen some definite improvements since we
reworked the way we're doing that function. It raises a question,
though--anyone else having weird performance issues using DateTime range
searches against DateTime values stored in the Catalog? We went from
checking a DateTime range to checking dates against an integer value, and
it's been working well.

 From: Chris McDonough [EMAIL PROTECTED]
 Date: Wed, 2 May 2001 12:08:19 -0400
 To: Tim McLaughlin [EMAIL PROTECTED], Chris Withers
 [EMAIL PROTECTED]
 Cc: Tim McLaughlin [EMAIL PROTECTED], [EMAIL PROTECTED]
 Subject: Re: [Zope-dev] Zope Server hanging :-(
 
 The docs (available via the --help switch) go into some of this detail, but
 I agree that a narrative explaining how to approach it from a functional
 perspective would be a good thing.
 
 
 - Original Message -
 From: Tim McLaughlin [EMAIL PROTECTED]
 To: 'Chris McDonough' [EMAIL PROTECTED]; Chris Withers
 [EMAIL PROTECTED]
 Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED]
 Sent: Wednesday, May 02, 2001 12:05 PM
 Subject: RE: [Zope-dev] Zope Server hanging :-(
 
 
 Nice info.  This could be useful for many, many things.  you oughtta put
 this in a how-to (if you haven't already).  thanks Chris.
 
 I'll use it if we have any more issues.  Currently, as I told ChrisW, the
 issues have stopped. But I'm still keeping my eye on it.
 
 -Original Message-
 From: Chris McDonough [mailto:[EMAIL PROTECTED]]
 Sent: Wednesday, May 02, 2001 12:00 PM
 To: Chris Withers
 Cc: Tim McLaughlin; [EMAIL PROTECTED]
 Subject: Re: [Zope-dev] Zope Server hanging :-(
 
 
 One of the more important bits is the active count when using
 the --detailed option to requestprofiler.  This tells you how many *other*
 requests were unfinished at the end of a particular request.
 
 So for instance, if you run requestprofiler like this:
 
 python requestprofiler.py log.file.name --detailed
 
 And you see a profile line like (header added for reference):
 
 startwin wout wend tot S   osize  code act url
 2001-04-19T01:19:420   120  12 E   26024  2005
 /Catalog/checkrep?P
 
 This means that:
 
 - this request started at 2001-04-19 1:19:42 UTC (start)
 
 - zserver waited 0 secs for input from the client (win)
 
 - zserver waited 12 secs for output from the publisher (wend)
 
 - the total time from start to end for the request was 12 secs (tot)
 
 - the request reached the end (E) stage (S, one of E, A, I,
 B)
 
 - the output size in bytes was 26024 (osize)
 
 - the http code was 200 (code)
 
 - the number of pending requests at the end of this request was 5
 (active)
 
 - the URL of the request (url)
 
 Hangs are also interesting.
 
 Use detailed reporting to sort by end stage:
 
 requestprofiler.py your.log.file --detailed --sort=endstage
 
 Any requests that don't show a stage of E for their endstage were hung
 requests.
 
 You can also use cumulative reporting to sort by hangs:
 
 requestprofiler.py your.log.file --cumulative --sort=hangs
 
 This will show the methods that hung most frequently.  Much of the time,
 you can find problems this way by seeing methods that enter an infinte
 loop
 or cause a segfault.
 
 Total time is also interesting:
 
 requestprofiler.py your.log.file --detailed --sort=total
 
 Shows you individual calls to methods that took a long time.
 
 - AND -
 
 requestprofiler.py your.log.file --cumulative --sort=total
 
 Shows you where most of the request time is spent.
 
 The goal is to indentify methods or requests that take a long time, don't
 return, or are accessed frequently.  If you can match this data up with
 specific problems you've experienced (possibly by way of times/dates), it
 becomes easier to form postulations about what is happening to Zope under
 load.  Then investigate those postulations by refactoring code or by doing
 more detailed logging in the places where you think the problems are.
 
 - C
 
 
 
 
 
 Hmmm, not really sure what I should be looking for :-S
 
 What ways should I look to run it and what should I do with the output?
 
 cheers,
 
 Chris
 
 
 
 
 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-05-02 Thread Michel Pelletier

On Wed, 2 May 2001, Chris McDonough wrote:

 The docs (available via the --help switch) go into some of this detail, but
 I agree that a narrative explaining how to approach it from a functional
 perspective would be a good thing.

This should go in the debuggin and testing chapter of the dev guide.
Wanna take a stab?

-Michel


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-29 Thread Erik Enge

On Fri, 27 Apr 2001, Chris Withers wrote:

 Well, I thought the upgrade to 2.3.2b2 would solve the problem but it
 hasn't :-( I have a (very important :-S) Zope instance which hangs at
 what seems like the slightest touch.

Maybe I can help.

I've seen similar behaviour, and by some chance I discovered what it was
that was causing it - which might not be the same thing that is causing
your problem.

When, in my Zope Python Product, I call a method that does not exist, Zope
dies, restarts and then is fine again.  If something (another method or a
browser perhaps) is calling this method again and again, Zope does a die,
restart, rinse, repeat.

I think the problem is that it doesn't throw the necessary Exception, or
that it actually throws it, but it is somehow captured by a try-except
that I've put out somewhere, so it doesn't show.

The problem is fixed once I stop calling the non-existant method.

I haven't had time to check whether this is the case with the latest
version of Zope, but it is with Zope 2.3.1b1, at least.

Maybe you have a method that is called when someone access that URL, but
that method does not exist?


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging: CST 0.8 didn't help :-(

2001-04-29 Thread Chris Withers

Joachim Werner wrote:
 
 Right now we don't face these problems any more. As far as I know, the only
 thing Stephan changed on the server was exchanging the CoreSession tracking
 by the newest version (0.8 I think), which had to be patched to run for us
 AFAIK. Now the server is reliable again.

Well, CST 0.8 worked out of the box for us, but it hasn't helped the problem.
(and we still get loads of those None has no attribute load errors ;-)

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging: 2.3.2b2 didn't help :-(

2001-04-29 Thread Chris Withers

Tim McLaughlin wrote:
 
 1.  Hangs on anything from Pythonscript or dtml

Sounds about right.

 2.  top shows no significant mem or cpu usage

Yup

 3.  threads are all unresponsive

yup

 4.  restart usually yields a .trX file (an aborted transaction I suppose)

Hadn't noticed, but I'll ook out for it.

 Any ideas?   Maybe it was something fixed in 2.3.1?  The causes seem to be
 somewhat different except that they all modify the ZODB

Yeah, likewise, but moving to 2.3.2b2 and updating all the BTree's involved
hasn't helped.

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-29 Thread Chris Withers

Chris McDonough wrote:
 
  I forgot to mention... our neither our z2.log or stupid_log show anything
  upon freezing.  afterwards the stupid_log shows the failed transaction
  cleanup, but that's it.
 
 Hmmm... it might be useful to turn on detailed request logging (-M logging)
 in the start file (see z2.py).

I'll give this a go next time I notice the thing has hung...

cheers,

Chris

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-29 Thread Dieter Maurer

Erik Enge writes:
  When, in my Zope Python Product, I call a method that does not exist, Zope
  dies, restarts and then is fine again.  If something (another method or a
  browser perhaps) is calling this method again and again, Zope does a die,
  restart, rinse, repeat.
  
  I think the problem is that it doesn't throw the necessary Exception, or
  that it actually throws it, but it is somehow captured by a try-except
  that I've put out somewhere, so it doesn't show.
Very strange! 

  Even if the exception was captured, this should not cause Zope to die.


Did you look in the log files? Have their been any core dumps?


Dieter

___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-29 Thread Erik Enge

On Sun, 29 Apr 2001, Dieter Maurer wrote:

 Very strange! 

Indeed.
 
 Did you look in the log files? Have their been any core dumps?

I'll try to do an hour or so of analysing this tomorrow, and I'll get back
to you.  :-)


___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-28 Thread Joachim Werner

 The symptoms are as follows:
 1. Go to a given URL, which doesn't respond (browser sits and spins)
 2. Doing a 'top' on the server shows python nowhere to be seen (so it's
not
 infinite-loop-processor-hogging)
 3. ps shows the python threads are all still there, but none of them will
 respond.
 3. doing a ./stop sucessfully stops Zope (the broswer returns a 'reset by
peer')
 4. doing a ./start yields the following the the stupid log file:

 Can anyone suggest how I might go about tracking this down? It's really
 beginning to piss me off. That said, I've seen quite a few posts now
complaining
 about Zope hanging and crashing, so I'm actually beginning to worry about
using
 Zope from a stability point of view.

These symptoms are indeed very similar to what we had on the
demo.iuveno-net.de site. We had an average up-time of around an hour or
so, and the system also didn't really crash but just hang ...

Right now we don't face these problems any more. As far as I know, the only
thing Stephan changed on the server was exchanging the CoreSession tracking
by the newest version (0.8 I think), which had to be patched to run for us
AFAIK. Now the server is reliable again.

I was really scared by this issue, too, as we had the same problems on a
demo server for a customer ...

Joachim.



___
Zope-Dev maillist  -  [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
**  No cross posts or HTML encoding!  **
(Related lists - 
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope )



Re: [Zope-dev] Zope Server hanging :-(

2001-04-27 Thread Chris McDonough



 I forgot to mention... our neither our z2.log or stupid_log show anything
 upon freezing.  afterwards the stupid_log shows the failed transaction
 cleanup, but that's it.

Hmmm... it might be useful to turn on detailed request logging (-M logging)
in the start file (see z2.py).

Attached is a script that I just checked into the trunk to do analysis of
the file generated by the -M log.  It can help you figure out if there's a
pattern to the hangs (whether it happens on a particular method, whether it
happens at heavy load time, whether it happens at a particular time of day,
etc.)  Do you think you'd be willing to play around with it a little bit to
try to discover a pattern?




 Tim

 -Original Message-
 From: Tim McLaughlin
 Sent: Friday, April 27, 2001 12:41 PM
 To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]'
 Subject: [Zope-dev] Zope Server hanging :-(


 Chris (and all),

 We seem to be having similar crashing issues.  And interestingly enough,
 they don't seem to show up on any of our servers except the 2.3.0 upgraded
 to 2.3.1.

 1.  Hangs on anything from Pythonscript or dtml
 2.  top shows no significant mem or cpu usage
 3.  threads are all unresponsive
 4.  restart usually yields a .trX file (an aborted transaction I suppose)

 Any ideas?   Maybe it was something fixed in 2.3.1?  The causes seem to
be
 somewhat different except that they all modify the ZODB

 Tim

 ___
 Zope-Dev maillist  -  [EMAIL PROTECTED]
 http://lists.zope.org/mailman/listinfo/zope-dev
 **  No cross posts or HTML encoding!  **
 (Related lists -
  http://lists.zope.org/mailman/listinfo/zope-announce
  http://lists.zope.org/mailman/listinfo/zope )



#!/usr/bin/env python
##
# 
# Zope Public License (ZPL) Version 1.0
# -
# 
# Copyright (c) Digital Creations.  All rights reserved.
# 
# This license has been certified as Open Source(tm).
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
# 
# 1. Redistributions in source code must retain the above copyright
#notice, this list of conditions, and the following disclaimer.
# 
# 2. Redistributions in binary form must reproduce the above copyright
#notice, this list of conditions, and the following disclaimer in
#the documentation and/or other materials provided with the
#distribution.
# 
# 3. Digital Creations requests that attribution be given to Zope
#in any manner possible. Zope includes a Powered by Zope
#button that is installed by default. While it is not a license
#violation to remove this button, it is requested that the
#attribution remain. A significant investment has been put
#into Zope, and this effort will continue if the Zope community
#continues to grow. This is one way to assure that growth.
# 
# 4. All advertising materials and documentation mentioning
#features derived from or use of this software must display
#the following acknowledgement:
# 
#  This product includes software developed by Digital Creations
#  for use in the Z Object Publishing Environment
#  (http://www.zope.org/).
# 
#In the event that the product being advertised includes an
#intact Zope distribution (with copyright and license included)
#then this clause is waived.
# 
# 5. Names associated with Zope or Digital Creations must not be used to
#endorse or promote products derived from this software without
#prior written permission from Digital Creations.
# 
# 6. Modified redistributions of any form whatsoever must retain
#the following acknowledgment:
# 
#  This product includes software developed by Digital Creations
#  for use in the Z Object Publishing Environment
#  (http://www.zope.org/).
# 
#Intact (re-)distributions of any official Zope release do not
#require an external acknowledgement.
# 
# 7. Modifications are encouraged but must be packaged separately as
#patches to official Zope releases.  Distributions that do not
#clearly separate the patches from the original work must be clearly
#labeled as unofficial distributions.  Modifications which do not
#carry the name Zope may be packaged in any form, as long as they
#conform to all of the clauses above.
# 
# 
# Disclaimer
# 
#   THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
#   EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
#   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
#   PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
#   CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
#   USE, DATA, OR PROFITS; OR BUSINESS