Re: [Zope-dev] Zope Server hanging :-(
On Wed, 6 Jun 2001 [EMAIL PROTECTED] wrote: ... at least then we know what the exception is. : Again - try that code in an interactive interpreter if you really want to find out what's going on... Yeah, thanks, that would work as a workaround, but isn't this buggish behaviour? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Wednesday 06 June 2001 18:03, Erik Enge wrote: On Wed, 6 Jun 2001 [EMAIL PROTECTED] wrote: ... at least then we know what the exception is. Again - try that code in an interactive interpreter if you really want to find out what's going on... Yeah, thanks, that would work as a workaround, but isn't this buggish behaviour? Absolutely - along with dozens of other places that Zope squashes exceptions. Anthony Baxter actually wrote a script that finds instances of bare except: clauses in the Zope source... http://www.zope.org/Members/anthony/BarewordExcepts Feel free to find the bad except: and submit a patch... Richard -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Wed, 6 Jun 2001, Richard Jones wrote: http://www.zope.org/Members/anthony/BarewordExcepts Feel free to find the bad except: and submit a patch... Ugh. There are tons of them... I'll see what I have time for. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Sun, 29 Apr 2001, Erik Enge wrote: I'll try to do an hour or so of analysing this tomorrow, and I'll get back to you. :-) Well, now we all know what vikings believe an hour or so mean, don't we? I figured it out, I think. Let's say I have these two methods: def a(): b() def b(): a() If I call a(), then Zope dies and restarts without giving me any error at all. Anyone got a clue? Makes debugging a bit tedious sometimes. I added a couple of print's in the methods and it seem to call them both many times before dying. I seem to remeber some ExessiveRecursion or some such error that popped up on me a year or two ago, has that left for happier hunting fields? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Erik Enge wrote: On Sun, 29 Apr 2001, Erik Enge wrote: I figured it out, I think. Let's say I have these two methods: def a(): b() def b(): a() If I call a(), then Zope dies and restarts without giving me any error at all. Anyone got a clue? There are many places in Zope where errors such as this (RuntimeError - try it in an interactive interpreter) are silently swallowed by too-broad except: clauses (except in this case, the exception is eaten, but the interpreter process exits anyway...). When we find that our code is dying without notice, we usually try to narrow down the area of code that could be to blame, and then wrap it in our own try/except like: try: do possibly bad stuff except: import traceback traceback.print_exc() raise ... at least then we know what the exception is. I seem to remeber some ExessiveRecursion or some such error that popped up on me a year or two ago, has that left for happier hunting fields? Again - try that code in an interactive interpreter if you really want to find out what's going on... Richard -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Erik Enge wrote: On Sun, 29 Apr 2001, Erik Enge wrote: I figured it out, I think. Let's say I have these two methods: def a(): b() def b(): a() If I call a(), then Zope dies and restarts without giving me any error at all. Anyone got a clue? This is an infinite recursion. I once had such a situation. Usually Python protects itself against such recursions by limiting the depth of its runtime stack. It raises a RuntimeError: runtime stack limit exceeded when its stack overflows. But in my case, the thread's runtime stack (maintained by the C runtime not Python) was more limited than the Python stack limit. When the thread's stack overflew, the process was killed by Solaris. Python did not have any chance to raise an exception as the death was immediate. We may need to ask the Python maintainers to increase the thread stack size or to more severely restrict the Python runtime stack. Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Dieter Maurer wrote: Erik Enge wrote: On Sun, 29 Apr 2001, Erik Enge wrote: I figured it out, I think. Let's say I have these two methods: def a(): b() def b(): a() If I call a(), then Zope dies and restarts without giving me any error at all. Anyone got a clue? This is an infinite recursion. I once had such a situation. Usually Python protects itself against such recursions by limiting the depth of its runtime stack. It raises a RuntimeError: runtime stack limit exceeded when its stack overflows. But in my case, the thread's runtime stack (maintained by the C runtime not Python) was more limited than the Python stack limit. When the thread's stack overflew, the process was killed by Solaris. Python did not have any chance to raise an exception as the death was immediate. We may need to ask the Python maintainers to increase the thread stack size or to more severely restrict the Python runtime stack. This can be done manually in your site.py - sys.setrecursionlimit() Richard -- Richard Jones [EMAIL PROTECTED] Senior Software Developer, Bizar Software (www.bizarsoftware.com.au) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging: DataPoint
Hmmm... saw these in the stupid log, just before the server crashed:
--
2001-05-03T10:09:34 INFO(0) Z2 CONFLICT Competing writes at,
/VirtualHostBase/http/server.nipltd.com:80/VirtualHostRoot/folder/object
Traceback (innermost last):
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
171, in publish
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/mapply.py, line 160,
in mapply
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
112, in call_object
snip lots of Python Scripts calling each other and DTML methods
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Connection.py, line 524,
in setstate
ConflictError: ('\\000\\000\\000\\000\\000\\020\\036\\304', 'extension class
Products.AProduct.Utility.AClass at 84ea878')
--
2001-05-03T10:09:42 INFO(0) Z2 CONFLICT Competing writes at,
/VirtualHostBase/http/server.nipltd.com:80/VirtualHostRoot/folder/object
Traceback (innermost last):
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZPublisher/Publish.py, line
175, in publish
File /usr/local/zope/Zope-2.3.2-src/lib/python/Zope/__init__.py, line 235, in
commit
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Transaction.py, line 300,
in commit
File /usr/local/zope/Zope-2.3.2-src/lib/python/ZODB/Connection.py, line 281,
in commit
ConflictError: '\000\000\000\000\000\020\037\245'
Could this bring it down? Should it? How can I track it down further?
cheers,
Chris
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging: DataPoint
Good morning, I presume? :-) Chris McDonough wrote: This probably isn't meaningful... unless it is. ;-) Huh? Conflict errors are normal... I wish they wouldn't be so big and have an UPPERCASE word in their description. OK, but I did notice them just before the server died. Maybe now would be a good time to run the analyser script. Unfortunately, I'd probably be expected to 'clean' the URLs of sensitive information. Hurm, d'ya think it'd be worth it? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging: DataPoint
Chris Withers wrote: Good morning, I presume? :-) Present! This probably isn't meaningful... unless it is. ;-) Huh? This is a Jim Fultonism. In English, it means this is probably not meaningful. Conflict errors are normal... I wish they wouldn't be so big and have an UPPERCASE word in their description. OK, but I did notice them just before the server died. Maybe now would be a good time to run the analyser script. Unfortunately, I'd probably be expected to 'clean' the URLs of sensitive information. Well the analyzer script only shows the first 20 or so chars of the URL by default unless you pass it the --verbose argument, so this might serve to do some cleaning. Also, getting rid of the URL altogether is something you can do by using Unix cut or on windows putting the results of the analyzer script into Excel and deleting the column with the URL in it. Hurm, d'ya think it'd be worth it? It's up to you, but I think so. - C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope Server hanging :-(
Sorry, Chris M, didn't mean to get you stuck in the documentation torture chamber ;-) Cheers, Tim -Original Message- From: Michel Pelletier [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 02, 2001 11:58 PM To: Chris McDonough Cc: Tim McLaughlin; Chris Withers; [EMAIL PROTECTED] Subject: Re: [Zope-dev] Zope Server hanging :-( On Wed, 2 May 2001, Chris McDonough wrote: The docs (available via the --help switch) go into some of this detail, but I agree that a narrative explaining how to approach it from a functional perspective would be a good thing. This should go in the debuggin and testing chapter of the dev guide. Wanna take a stab? -Michel ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Chris McDonough wrote: You can also use cumulative reporting to sort by hangs: requestprofiler.py your.log.file --cumulative --sort=hangs python requestprofiler.py my.log --cumulative --sort=hangs out2.txt resulted in: Traceback (innermost last): File requestprofiler.py, line 586, in ? analyze(open(sys.argv[1]), top, sortf, start, end, mode) File requestprofiler.py, line 346, in analyze write(dict, top) File requestprofiler.py, line 360, in write print str(stat)[:78] File requestprofiler.py, line 226, in __str__ body = ( File requestprofiler.py, line 273, in median else: return (all[i] + all[i2]) / 2 TypeError: number coercion failed Any ideas? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Please apply this patch, sorry!
*** requestprofiler.py 2001/04/26 02:51:49 1.3
--- requestprofiler.py 2001/05/03 15:12:13
***
*** 269,276
i2 = i + 1
v1 = all[i]
v2 = all[i2]
! if v1 == NA or v2 == NA: return I
! else: return (all[i] + all[i2]) / 2
def total(self):
t = 0
--- 269,276
i2 = i + 1
v1 = all[i]
v2 = all[i2]
! if type(v1) is type('') or type(v2) is type(''): return
I
! else: return (v1 + v2) / 2
def total(self):
t = 0
- Original Message -
From: Chris Withers [EMAIL PROTECTED]
To: Chris McDonough [EMAIL PROTECTED]
Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED]
Sent: Thursday, May 03, 2001 10:35 AM
Subject: Re: [Zope-dev] Zope Server hanging :-(
Chris McDonough wrote:
You can also use cumulative reporting to sort by hangs:
requestprofiler.py your.log.file --cumulative --sort=hangs
python requestprofiler.py my.log --cumulative --sort=hangs out2.txt
resulted in:
Traceback (innermost last):
File requestprofiler.py, line 586, in ?
analyze(open(sys.argv[1]), top, sortf, start, end, mode)
File requestprofiler.py, line 346, in analyze
write(dict, top)
File requestprofiler.py, line 360, in write
print str(stat)[:78]
File requestprofiler.py, line 226, in __str__
body = (
File requestprofiler.py, line 273, in median
else: return (all[i] + all[i2]) / 2
TypeError: number coercion failed
Any ideas?
cheers,
Chris
___
Zope-Dev maillist - [EMAIL PROTECTED]
http://lists.zope.org/mailman/listinfo/zope-dev
** No cross posts or HTML encoding! **
(Related lists -
http://lists.zope.org/mailman/listinfo/zope-announce
http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Thu, 3 May 2001, Chris McDonough wrote: It's no problem, especially as Michel had insomnia last night and did it. ;-) My best stuff comes out when I can't sleep. Those of you who are interested in looking at it, check out the ever-growing Zope Developer's Guide (shame on you if you haven't read it!). http://sourceforge.net/projects/zope-devel/ -Michel ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Thu, 3 May 2001, Michel Pelletier wrote: On Thu, 3 May 2001, Chris McDonough wrote: It's no problem, especially as Michel had insomnia last night and did it. ;-) My best stuff comes out when I can't sleep. Those of you who are interested in looking at it, check out the ever-growing Zope Developer's Guide (shame on you if you haven't read it!). Ok, well maybe not shame considering you have to check it out of CVS. ;0 -Michel ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Chris McDonough wrote: Absolutely! When you've got some representative data, and you've successfully run requestprofiler against it in various ways, let me know. Hmmm, not really sure what I should be looking for :-S What ways should I look to run it and what should I do with the output? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Attached is a script that I just checked into the trunk to do analysis of the file generated by the -M log. It can help you figure out if there's a pattern to the hangs (whether it happens on a particular method, whether it happens at heavy load time, whether it happens at a particular time of day, etc.) Do you think you'd be willing to play around with it a little bit to try to discover a pattern? Definitely, can you help me with 'reading the runes'? Absolutely! When you've got some representative data, and you've successfully run requestprofiler against it in various ways, let me know. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
One of the more important bits is the active count when using the --detailed option to requestprofiler. This tells you how many *other* requests were unfinished at the end of a particular request. So for instance, if you run requestprofiler like this: python requestprofiler.py log.file.name --detailed And you see a profile line like (header added for reference): startwin wout wend tot S osize code act url 2001-04-19T01:19:420 120 12 E 26024 2005 /Catalog/checkrep?P This means that: - this request started at 2001-04-19 1:19:42 UTC (start) - zserver waited 0 secs for input from the client (win) - zserver waited 12 secs for output from the publisher (wend) - the total time from start to end for the request was 12 secs (tot) - the request reached the end (E) stage (S, one of E, A, I, B) - the output size in bytes was 26024 (osize) - the http code was 200 (code) - the number of pending requests at the end of this request was 5 (active) - the URL of the request (url) Hangs are also interesting. Use detailed reporting to sort by end stage: requestprofiler.py your.log.file --detailed --sort=endstage Any requests that don't show a stage of E for their endstage were hung requests. You can also use cumulative reporting to sort by hangs: requestprofiler.py your.log.file --cumulative --sort=hangs This will show the methods that hung most frequently. Much of the time, you can find problems this way by seeing methods that enter an infinte loop or cause a segfault. Total time is also interesting: requestprofiler.py your.log.file --detailed --sort=total Shows you individual calls to methods that took a long time. - AND - requestprofiler.py your.log.file --cumulative --sort=total Shows you where most of the request time is spent. The goal is to indentify methods or requests that take a long time, don't return, or are accessed frequently. If you can match this data up with specific problems you've experienced (possibly by way of times/dates), it becomes easier to form postulations about what is happening to Zope under load. Then investigate those postulations by refactoring code or by doing more detailed logging in the places where you think the problems are. - C Hmmm, not really sure what I should be looking for :-S What ways should I look to run it and what should I do with the output? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Chris McDonough wrote: snip info Thanks, that's just what I was looking for The goal is to indentify methods or requests that take a long time, don't return, or are accessed frequently. Yup If you can match this data up with specific problems you've experienced (possibly by way of times/dates), well, I have tail -f of the detailed log constantly open on my desktop ;-) it becomes easier to form postulations about what is happening to Zope under load. To be honest, there really isn't any load on this server... maybe one or two people using it :-S Then investigate those postulations by refactoring code or by doing more detailed logging in the places where you think the problems are. Hmmm, I know what you mean but it doesn't sound like it's going to be fun :-S well, thanks for the info, I'll do what I can... Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Zope Server hanging :-(
Nice info. This could be useful for many, many things. you oughtta put this in a how-to (if you haven't already). thanks Chris. I'll use it if we have any more issues. Currently, as I told ChrisW, the issues have stopped. But I'm still keeping my eye on it. -Original Message- From: Chris McDonough [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 02, 2001 12:00 PM To: Chris Withers Cc: Tim McLaughlin; [EMAIL PROTECTED] Subject: Re: [Zope-dev] Zope Server hanging :-( One of the more important bits is the active count when using the --detailed option to requestprofiler. This tells you how many *other* requests were unfinished at the end of a particular request. So for instance, if you run requestprofiler like this: python requestprofiler.py log.file.name --detailed And you see a profile line like (header added for reference): startwin wout wend tot S osize code act url 2001-04-19T01:19:420 120 12 E 26024 2005 /Catalog/checkrep?P This means that: - this request started at 2001-04-19 1:19:42 UTC (start) - zserver waited 0 secs for input from the client (win) - zserver waited 12 secs for output from the publisher (wend) - the total time from start to end for the request was 12 secs (tot) - the request reached the end (E) stage (S, one of E, A, I, B) - the output size in bytes was 26024 (osize) - the http code was 200 (code) - the number of pending requests at the end of this request was 5 (active) - the URL of the request (url) Hangs are also interesting. Use detailed reporting to sort by end stage: requestprofiler.py your.log.file --detailed --sort=endstage Any requests that don't show a stage of E for their endstage were hung requests. You can also use cumulative reporting to sort by hangs: requestprofiler.py your.log.file --cumulative --sort=hangs This will show the methods that hung most frequently. Much of the time, you can find problems this way by seeing methods that enter an infinte loop or cause a segfault. Total time is also interesting: requestprofiler.py your.log.file --detailed --sort=total Shows you individual calls to methods that took a long time. - AND - requestprofiler.py your.log.file --cumulative --sort=total Shows you where most of the request time is spent. The goal is to indentify methods or requests that take a long time, don't return, or are accessed frequently. If you can match this data up with specific problems you've experienced (possibly by way of times/dates), it becomes easier to form postulations about what is happening to Zope under load. Then investigate those postulations by refactoring code or by doing more detailed logging in the places where you think the problems are. - C Hmmm, not really sure what I should be looking for :-S What ways should I look to run it and what should I do with the output? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
The docs (available via the --help switch) go into some of this detail, but I agree that a narrative explaining how to approach it from a functional perspective would be a good thing. - Original Message - From: Tim McLaughlin [EMAIL PROTECTED] To: 'Chris McDonough' [EMAIL PROTECTED]; Chris Withers [EMAIL PROTECTED] Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, May 02, 2001 12:05 PM Subject: RE: [Zope-dev] Zope Server hanging :-( Nice info. This could be useful for many, many things. you oughtta put this in a how-to (if you haven't already). thanks Chris. I'll use it if we have any more issues. Currently, as I told ChrisW, the issues have stopped. But I'm still keeping my eye on it. -Original Message- From: Chris McDonough [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 02, 2001 12:00 PM To: Chris Withers Cc: Tim McLaughlin; [EMAIL PROTECTED] Subject: Re: [Zope-dev] Zope Server hanging :-( One of the more important bits is the active count when using the --detailed option to requestprofiler. This tells you how many *other* requests were unfinished at the end of a particular request. So for instance, if you run requestprofiler like this: python requestprofiler.py log.file.name --detailed And you see a profile line like (header added for reference): startwin wout wend tot S osize code act url 2001-04-19T01:19:420 120 12 E 26024 2005 /Catalog/checkrep?P This means that: - this request started at 2001-04-19 1:19:42 UTC (start) - zserver waited 0 secs for input from the client (win) - zserver waited 12 secs for output from the publisher (wend) - the total time from start to end for the request was 12 secs (tot) - the request reached the end (E) stage (S, one of E, A, I, B) - the output size in bytes was 26024 (osize) - the http code was 200 (code) - the number of pending requests at the end of this request was 5 (active) - the URL of the request (url) Hangs are also interesting. Use detailed reporting to sort by end stage: requestprofiler.py your.log.file --detailed --sort=endstage Any requests that don't show a stage of E for their endstage were hung requests. You can also use cumulative reporting to sort by hangs: requestprofiler.py your.log.file --cumulative --sort=hangs This will show the methods that hung most frequently. Much of the time, you can find problems this way by seeing methods that enter an infinte loop or cause a segfault. Total time is also interesting: requestprofiler.py your.log.file --detailed --sort=total Shows you individual calls to methods that took a long time. - AND - requestprofiler.py your.log.file --cumulative --sort=total Shows you where most of the request time is spent. The goal is to indentify methods or requests that take a long time, don't return, or are accessed frequently. If you can match this data up with specific problems you've experienced (possibly by way of times/dates), it becomes easier to form postulations about what is happening to Zope under load. Then investigate those postulations by refactoring code or by doing more detailed logging in the places where you think the problems are. - C Hmmm, not really sure what I should be looking for :-S What ways should I look to run it and what should I do with the output? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
I'll vouch for the effectiveness of this method in tracking down some problems. Chris' advice helped us track down some horrendous numbers around a method calling DateTime. We've seen some definite improvements since we reworked the way we're doing that function. It raises a question, though--anyone else having weird performance issues using DateTime range searches against DateTime values stored in the Catalog? We went from checking a DateTime range to checking dates against an integer value, and it's been working well. From: Chris McDonough [EMAIL PROTECTED] Date: Wed, 2 May 2001 12:08:19 -0400 To: Tim McLaughlin [EMAIL PROTECTED], Chris Withers [EMAIL PROTECTED] Cc: Tim McLaughlin [EMAIL PROTECTED], [EMAIL PROTECTED] Subject: Re: [Zope-dev] Zope Server hanging :-( The docs (available via the --help switch) go into some of this detail, but I agree that a narrative explaining how to approach it from a functional perspective would be a good thing. - Original Message - From: Tim McLaughlin [EMAIL PROTECTED] To: 'Chris McDonough' [EMAIL PROTECTED]; Chris Withers [EMAIL PROTECTED] Cc: Tim McLaughlin [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Wednesday, May 02, 2001 12:05 PM Subject: RE: [Zope-dev] Zope Server hanging :-( Nice info. This could be useful for many, many things. you oughtta put this in a how-to (if you haven't already). thanks Chris. I'll use it if we have any more issues. Currently, as I told ChrisW, the issues have stopped. But I'm still keeping my eye on it. -Original Message- From: Chris McDonough [mailto:[EMAIL PROTECTED]] Sent: Wednesday, May 02, 2001 12:00 PM To: Chris Withers Cc: Tim McLaughlin; [EMAIL PROTECTED] Subject: Re: [Zope-dev] Zope Server hanging :-( One of the more important bits is the active count when using the --detailed option to requestprofiler. This tells you how many *other* requests were unfinished at the end of a particular request. So for instance, if you run requestprofiler like this: python requestprofiler.py log.file.name --detailed And you see a profile line like (header added for reference): startwin wout wend tot S osize code act url 2001-04-19T01:19:420 120 12 E 26024 2005 /Catalog/checkrep?P This means that: - this request started at 2001-04-19 1:19:42 UTC (start) - zserver waited 0 secs for input from the client (win) - zserver waited 12 secs for output from the publisher (wend) - the total time from start to end for the request was 12 secs (tot) - the request reached the end (E) stage (S, one of E, A, I, B) - the output size in bytes was 26024 (osize) - the http code was 200 (code) - the number of pending requests at the end of this request was 5 (active) - the URL of the request (url) Hangs are also interesting. Use detailed reporting to sort by end stage: requestprofiler.py your.log.file --detailed --sort=endstage Any requests that don't show a stage of E for their endstage were hung requests. You can also use cumulative reporting to sort by hangs: requestprofiler.py your.log.file --cumulative --sort=hangs This will show the methods that hung most frequently. Much of the time, you can find problems this way by seeing methods that enter an infinte loop or cause a segfault. Total time is also interesting: requestprofiler.py your.log.file --detailed --sort=total Shows you individual calls to methods that took a long time. - AND - requestprofiler.py your.log.file --cumulative --sort=total Shows you where most of the request time is spent. The goal is to indentify methods or requests that take a long time, don't return, or are accessed frequently. If you can match this data up with specific problems you've experienced (possibly by way of times/dates), it becomes easier to form postulations about what is happening to Zope under load. Then investigate those postulations by refactoring code or by doing more detailed logging in the places where you think the problems are. - C Hmmm, not really sure what I should be looking for :-S What ways should I look to run it and what should I do with the output? cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Wed, 2 May 2001, Chris McDonough wrote: The docs (available via the --help switch) go into some of this detail, but I agree that a narrative explaining how to approach it from a functional perspective would be a good thing. This should go in the debuggin and testing chapter of the dev guide. Wanna take a stab? -Michel ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Fri, 27 Apr 2001, Chris Withers wrote: Well, I thought the upgrade to 2.3.2b2 would solve the problem but it hasn't :-( I have a (very important :-S) Zope instance which hangs at what seems like the slightest touch. Maybe I can help. I've seen similar behaviour, and by some chance I discovered what it was that was causing it - which might not be the same thing that is causing your problem. When, in my Zope Python Product, I call a method that does not exist, Zope dies, restarts and then is fine again. If something (another method or a browser perhaps) is calling this method again and again, Zope does a die, restart, rinse, repeat. I think the problem is that it doesn't throw the necessary Exception, or that it actually throws it, but it is somehow captured by a try-except that I've put out somewhere, so it doesn't show. The problem is fixed once I stop calling the non-existant method. I haven't had time to check whether this is the case with the latest version of Zope, but it is with Zope 2.3.1b1, at least. Maybe you have a method that is called when someone access that URL, but that method does not exist? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging: CST 0.8 didn't help :-(
Joachim Werner wrote: Right now we don't face these problems any more. As far as I know, the only thing Stephan changed on the server was exchanging the CoreSession tracking by the newest version (0.8 I think), which had to be patched to run for us AFAIK. Now the server is reliable again. Well, CST 0.8 worked out of the box for us, but it hasn't helped the problem. (and we still get loads of those None has no attribute load errors ;-) cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging: 2.3.2b2 didn't help :-(
Tim McLaughlin wrote: 1. Hangs on anything from Pythonscript or dtml Sounds about right. 2. top shows no significant mem or cpu usage Yup 3. threads are all unresponsive yup 4. restart usually yields a .trX file (an aborted transaction I suppose) Hadn't noticed, but I'll ook out for it. Any ideas? Maybe it was something fixed in 2.3.1? The causes seem to be somewhat different except that they all modify the ZODB Yeah, likewise, but moving to 2.3.2b2 and updating all the BTree's involved hasn't helped. cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Chris McDonough wrote: I forgot to mention... our neither our z2.log or stupid_log show anything upon freezing. afterwards the stupid_log shows the failed transaction cleanup, but that's it. Hmmm... it might be useful to turn on detailed request logging (-M logging) in the start file (see z2.py). I'll give this a go next time I notice the thing has hung... cheers, Chris ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
Erik Enge writes: When, in my Zope Python Product, I call a method that does not exist, Zope dies, restarts and then is fine again. If something (another method or a browser perhaps) is calling this method again and again, Zope does a die, restart, rinse, repeat. I think the problem is that it doesn't throw the necessary Exception, or that it actually throws it, but it is somehow captured by a try-except that I've put out somewhere, so it doesn't show. Very strange! Even if the exception was captured, this should not cause Zope to die. Did you look in the log files? Have their been any core dumps? Dieter ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
On Sun, 29 Apr 2001, Dieter Maurer wrote: Very strange! Indeed. Did you look in the log files? Have their been any core dumps? I'll try to do an hour or so of analysing this tomorrow, and I'll get back to you. :-) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
The symptoms are as follows: 1. Go to a given URL, which doesn't respond (browser sits and spins) 2. Doing a 'top' on the server shows python nowhere to be seen (so it's not infinite-loop-processor-hogging) 3. ps shows the python threads are all still there, but none of them will respond. 3. doing a ./stop sucessfully stops Zope (the broswer returns a 'reset by peer') 4. doing a ./start yields the following the the stupid log file: Can anyone suggest how I might go about tracking this down? It's really beginning to piss me off. That said, I've seen quite a few posts now complaining about Zope hanging and crashing, so I'm actually beginning to worry about using Zope from a stability point of view. These symptoms are indeed very similar to what we had on the demo.iuveno-net.de site. We had an average up-time of around an hour or so, and the system also didn't really crash but just hang ... Right now we don't face these problems any more. As far as I know, the only thing Stephan changed on the server was exchanging the CoreSession tracking by the newest version (0.8 I think), which had to be patched to run for us AFAIK. Now the server is reliable again. I was really scared by this issue, too, as we had the same problems on a demo server for a customer ... Joachim. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Zope Server hanging :-(
I forgot to mention... our neither our z2.log or stupid_log show anything upon freezing. afterwards the stupid_log shows the failed transaction cleanup, but that's it. Hmmm... it might be useful to turn on detailed request logging (-M logging) in the start file (see z2.py). Attached is a script that I just checked into the trunk to do analysis of the file generated by the -M log. It can help you figure out if there's a pattern to the hangs (whether it happens on a particular method, whether it happens at heavy load time, whether it happens at a particular time of day, etc.) Do you think you'd be willing to play around with it a little bit to try to discover a pattern? Tim -Original Message- From: Tim McLaughlin Sent: Friday, April 27, 2001 12:41 PM To: '[EMAIL PROTECTED]'; '[EMAIL PROTECTED]' Subject: [Zope-dev] Zope Server hanging :-( Chris (and all), We seem to be having similar crashing issues. And interestingly enough, they don't seem to show up on any of our servers except the 2.3.0 upgraded to 2.3.1. 1. Hangs on anything from Pythonscript or dtml 2. top shows no significant mem or cpu usage 3. threads are all unresponsive 4. restart usually yields a .trX file (an aborted transaction I suppose) Any ideas? Maybe it was something fixed in 2.3.1? The causes seem to be somewhat different except that they all modify the ZODB Tim ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) #!/usr/bin/env python ## # # Zope Public License (ZPL) Version 1.0 # - # # Copyright (c) Digital Creations. All rights reserved. # # This license has been certified as Open Source(tm). # # Redistribution and use in source and binary forms, with or without # modification, are permitted provided that the following conditions are # met: # # 1. Redistributions in source code must retain the above copyright #notice, this list of conditions, and the following disclaimer. # # 2. Redistributions in binary form must reproduce the above copyright #notice, this list of conditions, and the following disclaimer in #the documentation and/or other materials provided with the #distribution. # # 3. Digital Creations requests that attribution be given to Zope #in any manner possible. Zope includes a Powered by Zope #button that is installed by default. While it is not a license #violation to remove this button, it is requested that the #attribution remain. A significant investment has been put #into Zope, and this effort will continue if the Zope community #continues to grow. This is one way to assure that growth. # # 4. All advertising materials and documentation mentioning #features derived from or use of this software must display #the following acknowledgement: # # This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/). # #In the event that the product being advertised includes an #intact Zope distribution (with copyright and license included) #then this clause is waived. # # 5. Names associated with Zope or Digital Creations must not be used to #endorse or promote products derived from this software without #prior written permission from Digital Creations. # # 6. Modified redistributions of any form whatsoever must retain #the following acknowledgment: # # This product includes software developed by Digital Creations # for use in the Z Object Publishing Environment # (http://www.zope.org/). # #Intact (re-)distributions of any official Zope release do not #require an external acknowledgement. # # 7. Modifications are encouraged but must be packaged separately as #patches to official Zope releases. Distributions that do not #clearly separate the patches from the original work must be clearly #labeled as unofficial distributions. Modifications which do not #carry the name Zope may be packaged in any form, as long as they #conform to all of the clauses above. # # # Disclaimer # # THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY # EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE # IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR # PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL DIGITAL CREATIONS OR ITS # CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, # SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT # LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF # USE, DATA, OR PROFITS; OR BUSINESS
