Re: [Zope] Zope hangs
Nico Grubert wrote at 2008-10-9 10:17 +0200:
> ...
>> If your system gets slow but still occasionally responds, you can
>> use "ZopeProfiler" to find out, where the time is spend.
>> Ensure to use "real time" as the timer (not cpu time).
>
>Hi Dieter
>
>thank you very much for the information.
>
>This morning the server did not respond and all 8 threads in the
>DeadlockDebugger's output look the same:
>
>Thread 1098918208
> ...
> File "/zope/intranet/Products/LDAPUserFolder/LDAPUserFolder.py", line
>1846, in _misc_cache
> return getResource('%s-misc_cache' % self._hash, SharedObject, ())
> File "/zope/intranet/Products/LDAPUserFolder/SharedResource.py", line
>50, in getResource
> _ResourceLock.release()
Your tracebacks really look strange: they almost all seem
to come from the release of looks.
Now, releasing a lock is a very fast operation.
An independent observer should rarely see the system in such
a state. That you see it there so often indicates that your
observer ("DeadlockDebugger") is not independent (we already knew that)
but in addition gets seriously confused -- thus, that we cannot trust
its observations when analysing the problem at hand.
You need to use a different observer.
As already written: if your Zope does not hang completely, "ZopeProfiler"
may be such an observer. It, too, is not independent, but different
things confuse it. Thus, we some luck, it may reveal the cause of
the current problems.
If your Zope hangs completely, then things get a bit more difficult.
You must attach the Zope process with a C level debugger. This
observer will not get confused by the Python GIL and is able
to observe threads that hold the GIL during long periods of time.
Unfortunately, quite a bit of preconditions must be met and
the analysis requires lots of knowledge about Python internas.
Therefore, I hope that you do not need an observer of this kind.
--
Dieter
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
> If your tracebacks are reliable, then neither of these entries > should cause "hanging" (apart from this: CPU usage of about 100 percent > does not indicate hanging but some other problem). > > Tracebacks do not need to be reliable. They can be unreliable when the > the "*.pyc/*.pyo" files do not correspond to the source. Therefore, > an initial step may be to delete all "*.pyc/*.pyo" files and try again. > > There is one case when "hanging" could apparently happen > in a "return": the operation before takes a long time and does not release > the GIL. Then, the deadlock debugger will get active only when the operation > finished -- and then sees an apparently harmless operation (such as a > "return"). > > We have had severe garbage collection problems (GC, too, holds the GIL > for a long time). To detect such problems, we have added a GIL monitor > to our system. This is an additional thread running in the system > which schedules itself at regular times and measures the difference > between the expected scheduling time and the actual activation time. > If the difference is too large, this is logged. > > > If your system gets slow but still occasionally responds, you can > use "ZopeProfiler" to find out, where the time is spend. > Ensure to use "real time" as the timer (not cpu time). Hi Dieter thank you very much for the information. This morning the server did not respond and all 8 threads in the DeadlockDebugger's output look the same: Thread 1098918208 File "/usr/local/Zope-2.10.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 25, in __init__ response=b) File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line 401, in publish_module environ, debug, request, response) File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line 202, in publish_module_standard response = publish(request, module_name, after_list, debug=debug) File "/zope/intranet/Products/PlacelessTranslationService/PatchStringIO.py", line 51, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line 119, in publish request, bind=1) File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/mapply.py", line 88, in mapply if debug is not None: return debug(object,args,context) File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line 42, in call_object result=apply(object,args) # Type s to step into published object. File "/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line 313, in __call__ return self._bindAndExec(args, kw, None) File "/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line 350, in _bindAndExec return self._exec(bound_data, args, kw) File "/usr/local/Zope-2.10.5/lib/python/Products/PythonScripts/PythonScript.py", line 327, in _exec result = f(*args, **kw) File "Script (Python)", line 20, in index_html File "/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line 313, in __call__ return self._bindAndExec(args, kw, None) File "/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line 350, in _bindAndExec return self._exec(bound_data, args, kw) File "/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/ZopePageTemplate.py", line 330, in _exec result = self.pt_render(extra_context=bound_names) File "/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/ZopePageTemplate.py", line 426, in pt_render result = PageTemplate.pt_render(self, source, extra_context) File "/zope/wwnet/Products/PTProfiler/ProfilerPatch.py", line 66, in __patched_render__ ret = self._patching_class._org_method(self, source, extra_context) File "/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/PageTemplate.py", line 89, in pt_render return super(PageTemplate, self).pt_render(c, source=source) File "/usr/local/Zope-2.10.5/lib/python/zope/pagetemplate/pagetemplate.py", line 117, in pt_render strictinsert=0, sourceAnnotations=sourceAnnotations)() File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 271, in __call__ self.interpret(self.program) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 346, in interpret handlers[opcode](self, args) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 891, in do_useMacro self.interpret(macro) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 346, in interpret handlers[opcode](self, args) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 949, in do_defineSlot self.interpret(slot) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 346, in interpret handlers[opcode](self, args) File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py", line 534, in d
Re: [Zope] Zope hangs
Nico Grubert wrote at 2008-10-2 10:13 +0200: > > These entries should not be the one that cause your problems > > (they are not computationally expensive). > > Do you see other entries as well? > >Thank you very much for your answer, Dieter. >There are 2 entries from in the DeadlockDebugger's output that may block >all other threads. If your tracebacks are reliable, then neither of these entries should cause "hanging" (apart from this: CPU usage of about 100 percent does not indicate hanging but some other problem). Tracebacks do not need to be reliable. They can be unreliable when the the "*.pyc/*.pyo" files do not correspond to the source. Therefore, an initial step may be to delete all "*.pyc/*.pyo" files and try again. There is one case when "hanging" could apparently happen in a "return": the operation before takes a long time and does not release the GIL. Then, the deadlock debugger will get active only when the operation finished -- and then sees an apparently harmless operation (such as a "return"). We have had severe garbage collection problems (GC, too, holds the GIL for a long time). To detect such problems, we have added a GIL monitor to our system. This is an additional thread running in the system which schedules itself at regular times and measures the difference between the expected scheduling time and the actual activation time. If the difference is too large, this is logged. If your system gets slow but still occasionally responds, you can use "ZopeProfiler" to find out, where the time is spend. Ensure to use "real time" as the timer (not cpu time). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
> import time
> T0 = time.time()
> conn.simple_bind_s(user_dn, user_pwd)
> T1 = time.time()
> open('/tmp/timings.log','a').write('%s\n' % (T1-T0))
Thanks, Peter.
I tried that and the max value for T1-T0 I got was: 0.162416934967
Regards
Nico
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
> These entries should not be the one that cause your problems
> (they are not computationally expensive).
> Do you see other entries as well?
Thank you very much for your answer, Dieter.
There are 2 entries from in the DeadlockDebugger's output that may block
all other threads.
Here is the full output of both entries:
1.
---
Thread 1098918208 (GET
/VirtualHostBase/http/intra:80/intranet/VirtualHostRoot/projects/index_DE):
File
"/usr/local/Zope-2.10.5/lib/python/ZServer/PubCore/ZServerPublisher.py",
line 25, in __init__
response=b)
File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line
401, in publish_module
environ, debug, request, response)
File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line
202, in publish_module_standard
response = publish(request, module_name, after_list, debug=debug)
File
"/zope/intranet/Products/PlacelessTranslationService/PatchStringIO.py",
line 51, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line
119, in publish
request, bind=1)
File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/mapply.py", line
88, in mapply
if debug is not None: return debug(object,args,context)
File "/usr/local/Zope-2.10.5/lib/python/ZPublisher/Publish.py", line
42, in call_object
result=apply(object,args) # Type s to step into published object.
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
313, in __call__
return self._bindAndExec(args, kw, None)
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
350, in _bindAndExec
return self._exec(bound_data, args, kw)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PythonScripts/PythonScript.py",
line 327, in _exec
result = f(*args, **kw)
File "Script (Python)", line 20, in index_html
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
313, in __call__
return self._bindAndExec(args, kw, None)
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
350, in _bindAndExec
return self._exec(bound_data, args, kw)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/ZopePageTemplate.py",
line 330, in _exec
result = self.pt_render(extra_context=bound_names)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/ZopePageTemplate.py",
line 426, in pt_render
result = PageTemplate.pt_render(self, source, extra_context)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/PageTemplate.py",
line 89, in pt_render
return super(PageTemplate, self).pt_render(c, source=source)
File
"/usr/local/Zope-2.10.5/lib/python/zope/pagetemplate/pagetemplate.py",
line 117, in pt_render
strictinsert=0, sourceAnnotations=sourceAnnotations)()
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 271, in __call__
self.interpret(self.program)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 346, in interpret
handlers[opcode](self, args)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 891, in do_useMacro
self.interpret(macro)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 346, in interpret
handlers[opcode](self, args)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 949, in do_defineSlot
self.interpret(slot)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 346, in interpret
handlers[opcode](self, args)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 534, in do_optTag_tal
self.no_tag(stuff[-2], stuff[-1])
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 516, in no_tag
self.interpret(program)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 346, in interpret
handlers[opcode](self, args)
File "/usr/local/Zope-2.10.5/lib/python/zope/tal/talinterpreter.py",
line 586, in do_setLocal_tal
self.engine.setLocal(name, self.engine.evaluateValue(expr))
File "/usr/local/Zope-2.10.5/lib/python/zope/tales/tales.py", line
696, in evaluate
return expression(self)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PageTemplates/ZRPythonExpr.py",
line 49, in __call__
return eval(self._code, vars, {})
File "PythonExpr", line 1, in
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
313, in __call__
return self._bindAndExec(args, kw, None)
File
"/usr/local/Zope-2.10.5/lib/python/Shared/DC/Scripts/Bindings.py", line
350, in _bindAndExec
return self._exec(bound_data, args, kw)
File
"/usr/local/Zope-2.10.5/lib/python/Products/PythonScripts/PythonScript.py",
line 3
Re: [Zope] Zope hangs
Nico Grubert wrote at 2008-9-30 11:49 +0200: >in the last days my Zope server seems to hang and the CPU is almost >every time at 100 %. When your CPU is almost at 100 % over an extended period, then you do not have a typical "hanging" problem but rather something like a busy loop. >I installed DeadlockDebugger and see a lot of these entries: > ... > File >"/usr/local/Zope-2.10.5/lib/python/ZODB/FileStorage/FileStorage.py", >line 562, in load > self._lock_release() > File "/usr/local/lib/python2.4/threading.py", line 109, in acquire > return rc These entries should not be the one that cause your problems (they are not computationally expensive). Do you see other entries as well? >- > >I am wondering what the last line > File "/usr/local/lib/python2.4/threading.py", line 109, in acquire > return rc >does exactly mean. Are the threads blocked? It is indeed strange that a "_lock_release" should end in an "acquire". Delete all "*.pyc" and "*.pyo" files and see whether these peculiarity disappear. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
2008/9/30 Nico Grubert <[EMAIL PROTECTED]>:
>> Yes, and you can use the Control_Panel/DebugInfo as well and scroll
>> down to the Connections table. Out of the 8 threads most of them
>> should be vacant (ie. None) and if there are some requests logged
>> there they should in the range of < 1 sec.
>> If any of them are longer than 1 sec they are either stuck or doing
>> something is supposed to take a long time (e.g. render a complex
>> report, do a big upload, etc.)
>>
>> I once had a similar problem and I hacked the LDAPUserFolder to do a
>> timing of the connections and found that connecting to the LDAP server
>> was super slow and thus each connection cost too much time and it
>> subsequently blocked the threads.
>
> Hi Peter
>
> thanks for your reply.
>
> At Control_Panel/DebugInfo I see
> ---
> None (44921)
> Tue Sep 30 13:24:18 2008 (0.37s)(... 'ACTUAL_URL':
> 'http://fog:8086/Control_Panel/DebugInfo/manage_main'}) (641)
> None(57523)
> None(69795)
> None(47637)
> Tue Sep 30 13:24:17 2008 (1.18s)(... path/to/any/doc')}) (48335)
> ---
>
> I sometimes see a LDAP call in the DeadlockProfiler output so it could
> be the reason.
>
> Do you still have the LDAP patch around? I'd like to do a
> timing of the connections, too.
>
It was work on at a client's site. But I think you should look for
somehing like this:
conn.simple_bind_s(user_dn, user_pwd) in LDAPDelegate.py
so it looks something like this:
import time
T0 = time.time()
conn.simple_bind_s(user_dn, user_pwd)
T1 = time.time()
open('/tmp/timings.log','a').write('%s\n' % (T1-T0))
> Regards
> Nico
> ___
> Zope maillist - Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
>
--
Peter Bengtsson,
work www.fry-it.com
home www.peterbe.com
hobby www.issuetrackerproduct.com
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
> Yes, and you can use the Control_Panel/DebugInfo as well and scroll > down to the Connections table. Out of the 8 threads most of them > should be vacant (ie. None) and if there are some requests logged > there they should in the range of < 1 sec. > If any of them are longer than 1 sec they are either stuck or doing > something is supposed to take a long time (e.g. render a complex > report, do a big upload, etc.) > > I once had a similar problem and I hacked the LDAPUserFolder to do a > timing of the connections and found that connecting to the LDAP server > was super slow and thus each connection cost too much time and it > subsequently blocked the threads. Hi Peter thanks for your reply. At Control_Panel/DebugInfo I see --- None (44921) Tue Sep 30 13:24:18 2008 (0.37s)(... 'ACTUAL_URL': 'http://fog:8086/Control_Panel/DebugInfo/manage_main'}) (641) None(57523) None(69795) None(47637) Tue Sep 30 13:24:17 2008 (1.18s)(... path/to/any/doc')}) (48335) --- I sometimes see a LDAP call in the DeadlockProfiler output so it could be the reason. Do you still have the LDAP patch around? I'd like to do a timing of the connections, too. Regards Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs
Yes, and you can use the Control_Panel/DebugInfo as well and scroll down to the Connections table. Out of the 8 threads most of them should be vacant (ie. None) and if there are some requests logged there they should in the range of < 1 sec. If any of them are longer than 1 sec they are either stuck or doing something is supposed to take a long time (e.g. render a complex report, do a big upload, etc.) I once had a similar problem and I hacked the LDAPUserFolder to do a timing of the connections and found that connecting to the LDAP server was super slow and thus each connection cost too much time and it subsequently blocked the threads. Peter 2008/9/30 Nico Grubert <[EMAIL PROTECTED]>: > Hi there > > in the last days my Zope server seems to hang and the CPU is almost > every time at 100 %. > > I installed DeadlockDebugger and see a lot of these entries: > - > File "/usr/local/Zope-2.10.5/lib/python/OFS/Traversable.py", line > 288, in unrestrictedTraverse > return obj > File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line > 767, in setstate > raise > File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line > 820, in _setstate > obj._p_serial = serial > File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 605, > in setGhostState > obj.__setstate__(state) > File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 597, > in getState > return unpickler.load() > File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 471, > in _persistent_load > return self.load_oid(reference) > File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 537, > in load_oid > return self._conn.get(oid) > File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line > 225, in get > return obj > File > "/usr/local/Zope-2.10.5/lib/python/ZODB/FileStorage/FileStorage.py", > line 562, in load > self._lock_release() > File "/usr/local/lib/python2.4/threading.py", line 109, in acquire > return rc > - > > I am wondering what the last line > File "/usr/local/lib/python2.4/threading.py", line 109, in acquire > return rc > does exactly mean. Are the threads blocked? > > > My configuration: > -Zope 2.10.5 running with 8 threads on a 2-CPU machine with 8 GB of RAM > -Python 2.4.4 > -LDAPUserfolder > -Silva CMS 2.0.8 > -ZODB size: ~6 Gbyte > -Only 6-10 people work in the CMS and no public access (spy bots, > crawlers) access the Zope server > > > Regards > Nico > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope hangs
Hi there in the last days my Zope server seems to hang and the CPU is almost every time at 100 %. I installed DeadlockDebugger and see a lot of these entries: - File "/usr/local/Zope-2.10.5/lib/python/OFS/Traversable.py", line 288, in unrestrictedTraverse return obj File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line 767, in setstate raise File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line 820, in _setstate obj._p_serial = serial File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 605, in setGhostState obj.__setstate__(state) File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 597, in getState return unpickler.load() File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load return self.load_oid(reference) File "/usr/local/Zope-2.10.5/lib/python/ZODB/serialize.py", line 537, in load_oid return self._conn.get(oid) File "/usr/local/Zope-2.10.5/lib/python/ZODB/Connection.py", line 225, in get return obj File "/usr/local/Zope-2.10.5/lib/python/ZODB/FileStorage/FileStorage.py", line 562, in load self._lock_release() File "/usr/local/lib/python2.4/threading.py", line 109, in acquire return rc - I am wondering what the last line File "/usr/local/lib/python2.4/threading.py", line 109, in acquire return rc does exactly mean. Are the threads blocked? My configuration: -Zope 2.10.5 running with 8 threads on a 2-CPU machine with 8 GB of RAM -Python 2.4.4 -LDAPUserfolder -Silva CMS 2.0.8 -ZODB size: ~6 Gbyte -Only 6-10 people work in the CMS and no public access (spy bots, crawlers) access the Zope server Regards Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
After I started Zope and make a few SQL queries, the Zope server shuts down with a "Segmentation fault". At the same time there is a log entry on my linux machine in /var/log/messages that reads: Jan 23 10:43:09 smut kernel: python[6231]: segfault at rip 0043c0d7 rsp 417f6ad0 error 4 From time to time Zope even hangs and eats all memory until the whole machine hangs completely. For the archive: The Python segfault was caused by the egenix mxODBC Database Adapter on a 64-Bit Linux. On a 32-Bit Linux everything works fine. Waiting for a patch for 64-Bit Linux now... Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs in FreeBSD with plenty of memory and CPU resources
My understanding is that get/setrecursionlimit is for the python internal limit and is not an indicator of the THREAD_STACK_SIZE value, but in any case, yes, we have compiled with that option and I don't think that these would be the symptoms I would see for a stack size issue anyway, but thanks for the suggestion! -- Jean On 2/26/07, [EMAIL PROTECTED] <[EMAIL PROTECTED]> wrote: Hmmm...unfortunately, that's not the cause of the problem (I'm responding since I'm on the East coast, Jean is on the West, and he's asleep right now): [EMAIL PROTECTED]: ~$ python Python 2.4.3 (#2, Jul 2 2006, 16:14:54) [GCC 3.4.2 [FreeBSD] 20040728] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. >>> import sys >>> sys.getrecursionlimit() 1000 Though I do appreciate you forwarding that URL along, because a *lot* of people have suggested the Python stack issue, and until now, we've been unable to definitively rule that out. It's almost as good to clear one problem as the potential cause as it is to fix things. :-) Alex Kirk > http://plone.org/documentation/faq/plone-on-bsd-python-stack > > Stefan > > > On 26. Feb 2007, at 06:06, Jean Lagarde wrote: > >> All, >> >> As a coincidence, that hang of Zope I mentioned in a message I posted >> yesterday has happened again today (second time), after a planned >> reboot. So here is a completely different issue from the memory one >> we've been discussing today. >> > -- > Anything that, in happening, causes something else to happen, > causes something else to happen. --Douglas Adams > > > ___ > Zope maillist - Zope@zope.org > http://mail.zope.org/mailman/listinfo/zope > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope-dev ) > ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs in FreeBSD with plenty of memory and CPU resources
Hmmm...unfortunately, that's not the cause of the problem (I'm responding since I'm on the East coast, Jean is on the West, and he's asleep right now): [EMAIL PROTECTED]: ~$ python Python 2.4.3 (#2, Jul 2 2006, 16:14:54) [GCC 3.4.2 [FreeBSD] 20040728] on freebsd5 Type "help", "copyright", "credits" or "license" for more information. import sys sys.getrecursionlimit() 1000 Though I do appreciate you forwarding that URL along, because a *lot* of people have suggested the Python stack issue, and until now, we've been unable to definitively rule that out. It's almost as good to clear one problem as the potential cause as it is to fix things. :-) Alex Kirk http://plone.org/documentation/faq/plone-on-bsd-python-stack Stefan On 26. Feb 2007, at 06:06, Jean Lagarde wrote: All, As a coincidence, that hang of Zope I mentioned in a message I posted yesterday has happened again today (second time), after a planned reboot. So here is a completely different issue from the memory one we've been discussing today. -- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs in FreeBSD with plenty of memory and CPU resources
http://plone.org/documentation/faq/plone-on-bsd-python-stack Stefan On 26. Feb 2007, at 06:06, Jean Lagarde wrote: All, As a coincidence, that hang of Zope I mentioned in a message I posted yesterday has happened again today (second time), after a planned reboot. So here is a completely different issue from the memory one we've been discussing today. -- Anything that, in happening, causes something else to happen, causes something else to happen. --Douglas Adams ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope hangs in FreeBSD with plenty of memory and CPU resources
All, As a coincidence, that hang of Zope I mentioned in a message I posted yesterday has happened again today (second time), after a planned reboot. So here is a completely different issue from the memory one we've been discussing today. There was other weirdness (for whatever reason _mysql could not find libmysqlclient_r.so.15 on some restart attempts), but here is something that seemed to occur consistently while Zope was unresponsive: On FreeBSD 5.4, with zserver-threads set to the default 4, there were seven Zope threads with the following states (CPU use is 0% and memory use is well below the maximum, pretty much where it is at after a normal fresh Zope restart): select ksesig kserel kqread kqread kqread kqread When Zope is operating normally and idle, I generally only see the three top threads, and when serving pages, I rarely see more than three threads; one actually often disappears and I only see two, one in the RUN state. I know that the kqread state has to do with the kqueue or kevent system call, and assume that Zope being in that waiting state for a long time is abnormal. Asking here in case someone has seen this behavior before and knows what might be the cause. As for the last and only other time this happened, Zope regained normal behavior by itself after a little while (in the order of half an hour). I don't see anything abnormal in the event log (other than the _mysql issue I mentioned above), and while stuck, requests do not register in the Z2.log. Browsers do get a connection because they state "waiting for..." but they do not receive anything, not even headers. After a fresh restart, I could get Zope to respond to a relatively benign request (the top level ZMI page), while there were only three Zope threads, but if asking for more, Zope would quickly get to the unresponsive state described above. We're thinking that we may be having underlying OS issues and will probably ask on a FreeBSD list and/or upgrade FreebBSD and hope that helps, but I'm asking here first just in case that this pattern of behavior was observed and resolved before. Cheers, Jean ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
Nico Grubert wrote at 2007-2-9 08:56 +0100: > ... >After I started Zope and make a few SQL queries, the Zope server shuts >down with a "Segmentation fault". >At the same time there is a log entry on my linux machine in >/var/log/messages that reads: >Jan 23 10:43:09 smut kernel: python[6231]: segfault at >rip 0043c0d7 rsp 417f6ad0 error 4 > > From time to time Zope even hangs and eats all memory until the whole >machine hangs completely. This seems to be a different problem (so it might have a common cause with the SIGSEGV problem). -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
Hi Nico, I've also struggled with Easysoft's OOB (Odbc-Odbc Bridge) and with dire results. The main problem we had wasn't seg faults but that it got hung up and stopped responding probably due to abandonned closed connection sockets. On a completely different customer I managed to get a Zope 2.8.8 recently working with egenix mxODBC Zope DA to a MS SQL Server using FreeTDS and that worked a lot better. I was happy to this time avoid the extra headache of Easysoft OOB. NB! I couldn't get it to work with iODBC. It only worked with unixODBC. On another project where we used Easysoft I did some rough profiling on the speed difference between unixODBC and iODBC and found no distinct difference so if you, like me, can only get unixODBC to talk to FreeTDS then don't worry about iODBC. Nico Grubert wrote: I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of RAM) with Python 2.4.3. installed. From time to time, Zope hangs and I cannot access it anymore. I tried to use the "Zope DeadlockDebugger" but if my Zope hangs I cannot call the URL "http://myzopesite:8080/manage_debug_threads?secret_password"; to let Zope DeadlockDebugger show any useful information. Zope does not response. If Zope hangs, the python process eats all the memory and the machine starts to swap. I think I have found the reason for this problem: The problem comes from a python segmentation fault caused by either "egenix mxODBCDA" database adapter or "Easysoft ODBC-ODBC Bridge". Here is a short description about the problem: --- From a linux machine I am accessing a remote windows 2003 server machine to query a Filemaker Pro database. I have "Easysoft ODBC-ODBC Server 2.0.6 for Windows" installed in my windows machine. On my linux machine I use the "unixODBC" (Easysoft ODBC-ODBC Client 2.0.6 for Linux) driver and the Zope product "mxODBCZopeDA" to query the Filemaker Database. After I started Zope and make a few SQL queries, the Zope server shuts down with a "Segmentation fault". At the same time there is a log entry on my linux machine in /var/log/messages that reads: Jan 23 10:43:09 smut kernel: python[6231]: segfault at rip 0043c0d7 rsp 417f6ad0 error 4 From time to time Zope even hangs and eats all memory until the whole machine hangs completely. Every time this happen, the last entry in /log/trace.log is a request to a Zope Page Template where a ZSQL method is called to query the ODBC source (Filemaker). At the same time there is a "python segmentation fault" log entry on my linux machine in /var/log/messages. One of the developers of the "mxODBCZopeDA" product told me the error is definitely coming from the ODBC-ODBC bridge. However, I can query the Filemaker database and get nice results but after a while my Zope server shuts down with the error messages above. I don't do any SQL INSERT or SQL UPDATE. I just do some SQL "SELECT...FROM...WHERE..." queries. My setup: + 64-Bit Suse Linux 10.1 Professional + unixODBC & unixODBC-devel 2.2.11 + Easysoft ODBC-ODBC Client 2.0.6 for Linux + Easysoft ODBC-ODBC Server 2.0.6 for Windows + ODBC Source: Filemaker Pro 8 Advanced Server on a Windows2003 Server + Python 2.4.3. + Zope 2.9.6. + mxODBCZopeDA 1.0.10 On a 32-Bit Suse 9.2 with unixODBC & unixODBC-devel 2.2.9, Python 2.5.3. and Zope 2.8.8. installed this error never occured. The segmentation fault only occurs on my 64 bit OS as described above. I'm waiting for an answer from the Easysoft ODBC-ODBC support but I'm gonna try to update Python 2.4.3. to 2.4.5. in order to see if it may solve the problem. Thanks to Dieter, Suresh and Paul. Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of RAM) with Python 2.4.3. installed. From time to time, Zope hangs and I cannot access it anymore. I tried to use the "Zope DeadlockDebugger" but if my Zope hangs I cannot call the URL "http://myzopesite:8080/manage_debug_threads?secret_password"; to let Zope DeadlockDebugger show any useful information. Zope does not response. If Zope hangs, the python process eats all the memory and the machine starts to swap. I think I have found the reason for this problem: The problem comes from a python segmentation fault caused by either "egenix mxODBCDA" database adapter or "Easysoft ODBC-ODBC Bridge". Here is a short description about the problem: --- From a linux machine I am accessing a remote windows 2003 server machine to query a Filemaker Pro database. I have "Easysoft ODBC-ODBC Server 2.0.6 for Windows" installed in my windows machine. On my linux machine I use the "unixODBC" (Easysoft ODBC-ODBC Client 2.0.6 for Linux) driver and the Zope product "mxODBCZopeDA" to query the Filemaker Database. After I started Zope and make a few SQL queries, the Zope server shuts down with a "Segmentation fault". At the same time there is a log entry on my linux machine in /var/log/messages that reads: Jan 23 10:43:09 smut kernel: python[6231]: segfault at rip 0043c0d7 rsp 417f6ad0 error 4 From time to time Zope even hangs and eats all memory until the whole machine hangs completely. Every time this happen, the last entry in /log/trace.log is a request to a Zope Page Template where a ZSQL method is called to query the ODBC source (Filemaker). At the same time there is a "python segmentation fault" log entry on my linux machine in /var/log/messages. One of the developers of the "mxODBCZopeDA" product told me the error is definitely coming from the ODBC-ODBC bridge. However, I can query the Filemaker database and get nice results but after a while my Zope server shuts down with the error messages above. I don't do any SQL INSERT or SQL UPDATE. I just do some SQL "SELECT...FROM...WHERE..." queries. My setup: + 64-Bit Suse Linux 10.1 Professional + unixODBC & unixODBC-devel 2.2.11 + Easysoft ODBC-ODBC Client 2.0.6 for Linux + Easysoft ODBC-ODBC Server 2.0.6 for Windows + ODBC Source: Filemaker Pro 8 Advanced Server on a Windows2003 Server + Python 2.4.3. + Zope 2.9.6. + mxODBCZopeDA 1.0.10 On a 32-Bit Suse 9.2 with unixODBC & unixODBC-devel 2.2.9, Python 2.5.3. and Zope 2.8.8. installed this error never occured. The segmentation fault only occurs on my 64 bit OS as described above. I'm waiting for an answer from the Easysoft ODBC-ODBC support but I'm gonna try to update Python 2.4.3. to 2.4.5. in order to see if it may solve the problem. Thanks to Dieter, Suresh and Paul. Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
Nico Grubert wrote at 2007-1-15 13:22 +0100: >I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of >RAM) with Python 2.4.3. installed. > From time to time, Zope hangs and I cannot access it anymore. I tried >to use the "Zope DeadlockDebugger" but if my Zope hangs I cannot call >the URL "http://myzopesite:8080/manage_debug_threads?secret_password"; to >let Zope DeadlockDebugger show any useful information. Zope does not >response. >If Zope hangs, the python process eats all the memory and the machine >starts to swap. Then, this is a case where you need debugging on C level, e.g. with "GDB". You need a Python version with debug symbols (those installations that come with the OS distribution usually lack them, you need to build your Python yourself -- and with it your Zope). Then you attach the Zope process (the top level process, in case that threads show as processes as well) when the problem occurs and analyse its threads -- especially the one that is running when you attached. If the problem occurs in Python (which is unlikely), then I can provide you with GDB command definitions to analyse the Python call chain (the Python traceback) from the C-level backtrace. > ... >These kind of Conflict errors occur almost every minute. >It might be that these errors have something to do with it but I am not >sure. This is another problem -- probably unrelated to the hangs. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs and Python eats all memory
On Mon, Jan 15, 2007 at 01:22:39PM +0100, Nico Grubert wrote: > Dear list members, > > I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of > RAM) with Python 2.4.3. installed. > From time to time, Zope hangs and I cannot access it anymore. I tried > to use the "Zope DeadlockDebugger" but if my Zope hangs I cannot call > the URL "http://myzopesite:8080/manage_debug_threads?secret_password"; to > let Zope DeadlockDebugger show any useful information. Zope does not > response. > If Zope hangs, the python process eats all the memory and the machine > starts to swap. > > The "top" command in the shell tells me: > - > Tasks: 91 total, 2 running, 89 sleeping, 0 stopped, 0 zombie > Cpu(s): 10.4%us, 0.4%sy, 0.0%ni, 89.0%id, 0.2%wa, 0.0%hi, 0.0%si > Mem: 9041256k total, 9025124k used,16132k free,10604k buffers > Swap: 4208988k total, 4208988k used,0k free, 9472k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND > > 29992 wwwrun16 0 12.0g 8.3g 2860 S 19 95.9 3:52.66 python > - > > There are several add-ons for Zope installed, like: > - "Psycopg" Postgres Database Adapter > - "mxODBCDA" ODBC Database Adapter > - LDAPUserfolder > - "Silva" Content Management System > - PIL > > Furthermore, I see a lot of Conflict Errors in the "event.log" like e.g.: > - > ZPublisher.Conflict ConflictError at > /VirtualHostBase/http/193.134.202.20:80/mysite/VirtualHostRoot/: > database conflict error (oid 0x0435, class BTrees._OOBTree.OOBTree, > serial this txn started with 0x036aeb14ab1c4b88 2007-01-15 > 12:04:40.104030, serial currently committed 0x036aeb1d0cc5d099 > 2007-01-15 12:13:02.993605) (80 conflicts (0 unresolved) since startup > at Mon Jan 15 11:44:55 2007) > - > > These kind of Conflict errors occur almost every minute. > It might be that these errors have something to do with it but I am not > sure. > > How can I figure out what exactly causes Zope to hang? > If you need more information, please let me know. > > Thanks in advance... > > Regards, > Nico It sounds to me like you are simply experiencing "swap death". Zope isn't really hung - it's just using so much memory that you're "forever" waiting on disk I/O to handle all those pages. The system should recover eventually if there's no more load; but unfortunately the typical case is that once you start swapping badly, user requests keep piling up in the queue, so things only continue to get worse. I'm impressed at your stats though. I've never had a box with 8 GB of RAM, much less got Zope to use up all of it :-) Have you tried the "debug spinning zope" recipe? http://www.zopelabs.com/cookbook/1073504990 that might give you a clue how you got into this state. unfortunately since the whole system is swapping like crazy, working at the shell is probably no fun either :-) P.S. How's your SU700 treating you? I sold mine years ago :) -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope hangs and Python eats all memory
Dear list members, I am running Zope 2.9.6. on a 64-Bit Suse Linux 10.1 machine (9 Gbyte of RAM) with Python 2.4.3. installed. From time to time, Zope hangs and I cannot access it anymore. I tried to use the "Zope DeadlockDebugger" but if my Zope hangs I cannot call the URL "http://myzopesite:8080/manage_debug_threads?secret_password"; to let Zope DeadlockDebugger show any useful information. Zope does not response. If Zope hangs, the python process eats all the memory and the machine starts to swap. The "top" command in the shell tells me: - Tasks: 91 total, 2 running, 89 sleeping, 0 stopped, 0 zombie Cpu(s): 10.4%us, 0.4%sy, 0.0%ni, 89.0%id, 0.2%wa, 0.0%hi, 0.0%si Mem: 9041256k total, 9025124k used,16132k free,10604k buffers Swap: 4208988k total, 4208988k used,0k free, 9472k cached PID USER PR NI VIRT RES SHR S %CPU %MEMTIME+ COMMAND 29992 wwwrun16 0 12.0g 8.3g 2860 S 19 95.9 3:52.66 python - There are several add-ons for Zope installed, like: - "Psycopg" Postgres Database Adapter - "mxODBCDA" ODBC Database Adapter - LDAPUserfolder - "Silva" Content Management System - PIL Furthermore, I see a lot of Conflict Errors in the "event.log" like e.g.: - ZPublisher.Conflict ConflictError at /VirtualHostBase/http/193.134.202.20:80/mysite/VirtualHostRoot/: database conflict error (oid 0x0435, class BTrees._OOBTree.OOBTree, serial this txn started with 0x036aeb14ab1c4b88 2007-01-15 12:04:40.104030, serial currently committed 0x036aeb1d0cc5d099 2007-01-15 12:13:02.993605) (80 conflicts (0 unresolved) since startup at Mon Jan 15 11:44:55 2007) - These kind of Conflict errors occur almost every minute. It might be that these errors have something to do with it but I am not sure. How can I figure out what exactly causes Zope to hang? If you need more information, please let me know. Thanks in advance... Regards, Nico ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs when database takes too long to respond
On 9/7/06, Maciej Wisniowski <[EMAIL PROTECTED]> wrote: > When zope "hangs" you may take a look at your console or > file to see what is the state of threads. Ergh.. I should have written that you'll see whether threads are waiting for database... If any thread does something different it won't be shown here. If there are any threads left to serve web requests, you could look at the Control Panel debug information, where you can see the currently processing requests. -- Martijn Pieters ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs when database takes too long to respond
When zope "hangs" you may take a look at your console or file to see what is the state of threads. Ergh.. I should have written that you'll see whether threads are waiting for database... If any thread does something different it won't be shown here. -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs when database takes too long to respond
Besides DeadlockDebugger, is there a way to see just how many threads Zope is currently using ? What do you mean by 'currently using'? Zope instance always uses the number of threads you've specified in zope.conf. You may try 'ps -aux|grep zope' if you're on linux box. If you want to see what threads are doing then I don't know other tool than DeadlockDebugger for this. Or... the manual solution... you may edit a method in ZMySQLDA (possibly query in db.py) to print some statements before and after it queries the database. Eg. import thread print 'before executing query. thread id: %s' % (thread.get_ident()) r = c.execute() # or something that queries db print 'after executing query. thread id: %s' % (thread.get_ident()) I don't remember exactly how to display thread id so check this because the code above may be broken! You should start your zope in console with runzope or runzope.bat - you'll she statements from db.py. Of course you may write these to files with logging module or something if you don't want to use runzope. When zope "hangs" you may take a look at your console or file to see what is the state of threads. HTH -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope hangs when database takes too long to respond
> Seems that all your Zope threads are busy (they're waiting for database response). There is no more threads to serve next requests so they're waiting (you may say that zope hangs in this situation, but in fact all it's threads are waiting for DB). You may install DeadlockDebugger product and you'll be able to see what your's Zope threads are really doing. Thanks. Besides DeadlockDebugger, is there a way to see just how many threads Zope is currently using ? Regards, Danny ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs when database takes too long to respond
Whenever the MySQL server experiences heavy load, Zope seems to hang, such that we can't even access the ZMI. Is this a bug in version 2.7.6 or ZMySQLDA ? Seems that all your Zope threads are busy (they're waiting for database response). There is no more threads to serve next requests so they're waiting (you may say that zope hangs in this situation, but in fact all it's threads are waiting for DB). You may install DeadlockDebugger product and you'll be able to see what your's Zope threads are really doing. Or are there some settings I can tweak to adjust some database connection timeout value ? AFAIK not in Zope. Take a look at ZMySQLDA or rather the python MySQL library it uses. Maybe there is something like that. You may ask this question on Zope-DB list. -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
RE: [Zope] Zope hangs when database takes too long to respond
Sorry for the typo error. The subject should read "hangs", not "hands" . From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Sinang, DannySent: Wednesday, September 06, 2006 10:56 PMTo: zope@zope.orgSubject: [Zope] Zope hands when database takes too long to respond Hello, We wrote an application (on Zope v2.7.6-final) that queries a MySQL database via ZMySQLDA v2.0.8. Whenever the MySQL server experiences heavy load, Zope seems to hang, such that we can't even access the ZMI. Is this a bug in version 2.7.6 or ZMySQLDA ? Or are there some settings I can tweak to adjust some database connection timeout value ? Regards, Danny ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope hangs with 'idle in transaction'
Yuan HOng wrote at 2005-10-27 18:18 +0800: > ... >I have a site running with Zope 2.7.6 under Linux. The site access a >PostgreSQL database through ZSQL and psycopg 1.1.18. Only read access >is used, nothing is written to the database. > >The one thing that annoys me very much is, with time passing, Zope >will eventually hang and has to be restarted. When I use ps -Af to >view the data connection status, I discovered that whenever Zope >hangs, I can found all zope database connection in the status 'idle in >transaction', like below: > >postgres 13605 799 0 16:22 ?00:00:07 postgres: zopeapp >backupdb 127.0.0.1 idle in transaction >postgres 13611 799 0 16:22 ?00:00:06 postgres: zopeapp >backupdb 127.0.0.1 idle in transaction >postgres 13614 799 0 16:22 ?00:00:00 postgres: zopeapp >backupdb 127.0.0.1 idle in transaction >postgres 14078 799 0 16:26 ?00:00:05 postgres: zopeapp >backupdb 127.0.0.1 idle in transaction I saw something like this when more than a single database connection was used in the same transaction -- it is a typical database deadlock: some connection locked some resource and later on in the transaction a second connection tried to access the locked resource -- deadlock. Avoid using more than one conntection to the same database in a single transaction! -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope hangs with 'idle in transaction'
Hi, list, I have a site running with Zope 2.7.6 under Linux. The site access a PostgreSQL database through ZSQL and psycopg 1.1.18. Only read access is used, nothing is written to the database. The one thing that annoys me very much is, with time passing, Zope will eventually hang and has to be restarted. When I use ps -Af to view the data connection status, I discovered that whenever Zope hangs, I can found all zope database connection in the status 'idle in transaction', like below: postgres 13605 799 0 16:22 ?00:00:07 postgres: zopeapp backupdb 127.0.0.1 idle in transaction postgres 13611 799 0 16:22 ?00:00:06 postgres: zopeapp backupdb 127.0.0.1 idle in transaction postgres 13614 799 0 16:22 ?00:00:00 postgres: zopeapp backupdb 127.0.0.1 idle in transaction postgres 14078 799 0 16:26 ?00:00:05 postgres: zopeapp backupdb 127.0.0.1 idle in transaction It seems zope is initiating some database transactions even though my site only access the database for reading. And those transaction somehow get into deadlock status and finally used up the maximun db connection allowed for zope. But why? How do I find out what zope is doing and why the connection are locked up? Any clue and direction pointing are heartily appreciated. Regards, -- Hong Yuan 大管家网上建材超市 装修装潢建材一站式购物 http://www.homemaster.cn ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope "hangs" on Add Document
Hello, I have this problem, it happened for the first time & I can't explain it. I create a folder under the root zope folder. In this I want to add a dtml method (or document). I get the add form, enter an id and a title, click "add" and zope just does... nothing. Anybody got any idea what this could be? I use apache w/proxypass, Zope 2.1.6 on Linux on a celeron 300 with 192 MB of RAM, and I have not changed a thing since I last used zope. Ideas? Guesses? A simple restart doesn't help a bit. Best wishes, Nils -- http://nils.jeppe.de/ +49 177 7369365 ___ Zope maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
