[Zope] zope unresponsive
I know that there is a switch between zeo and zope and probably a firewall too, but how do I prove this is the problem. This is on production server in a military installation. I have major problems getting any kind of trouble shooting support. First we don't get access, and second no kind of debugging is allowed. You couldn't imagine the paperwork and the three months it took for me to get deadlockdebugger installed. Thanks, Paul ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] zope unresponsive
I have posted this several times, but have not until now been able to
get DeadlockDebugger installed. I see several people have had this
problem, but no-one has posted a solution.
zope 2.9.5 + zeo
pythonm2.4.3
Red Hat RHEL 4
Plone 2.5.1
Our zeo clients hang intermittently. We have no way of reproducing the
problem, but it occurs daily. The client hangs and a restart seems to
fix the
problem.
In the event log with tracing on we get
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
There are hundreds to thousands of these until the server is restarted.
In the zeo log we get
Error caught in asyncor asyncore.py
error:(110,'Connection timed out')
We have been trying to track this down and have had no luck. Does
anyone have
any suggestions? Below is our deadlock debugger output
Threads traceback dump at 2007-02-23 15:26:50
Thread -1269564496 (GET
/VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training):
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC
ore/ZServerPublisher.py, line 23, in __init__
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py, line 395, in publish_module
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py, line 196, in publish_module_standard
File
/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat
chStringIO.py, line 34, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py, line 115, in publish
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m
apply.py, line 88, in mapply
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py, line 41, in call_object
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc
ripts/Bindings.py, line 311, in __call__
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc
ripts/Bindings.py, line 348, in _bindAndExec
File
/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py,
line 195, in _exec
result = self.pt_render(extra_context=bound_names)
File
/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py, line
38, in FSPT_pt_render
result = FSPageTemplate.inheritedAttribute('pt_render')(
File
/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py, line
92, in PT_pt_render
tal=not source, strictinsert=0)()
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 238, in __call__
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 281, in interpret
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 749, in do_useMacro
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 281, in interpret
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 457, in do_optTag_tal
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 442, in do_optTag
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 437, in no_tag
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 281, in interpret
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 749, in do_useMacro
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 281, in interpret
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py, line 507, in do_setLocal_tal
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/TALES.py, line 221, in evaluate
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py, line 185, in __call__
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py, line 180, in _eval
File
/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py, line 85, in render
File
/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py,
line 66, in globalize
self._initializeData(options=options)
File
/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py,
line 147, in _initializeData
self._data['language'] = self.request.get('language', None) or \
File /apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py,
line 58, in generatedAccessor
return schema[name].get(self, **kw)
File /apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py,
line 802, in get
value = ObjectField.get(self, instance, **kwargs)
File
Re: [Zope] zope unresponsive
On 2/24/07, Paul Williams [EMAIL PROTECTED] wrote: I have posted this several times, but have not until now been able to get DeadlockDebugger installed. I see several people have had this problem, but no-one has posted a solution. zope 2.9.5 + zeo pythonm2.4.3 Red Hat RHEL 4 Plone 2.5.1 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. I don't know if this even relevant but I also have had Zope go into lala land and struggled for a while trying to figure out what was going on. Like your situation it happened intermittently and a reboot recovered (and sometimes seemed to recover on its own). And like you I tried various methods to spot the problem including installing DeadlockDebugger. This was in various versions of Zope running on Win2003. In the end I found it was a page/browser at the root of the problem. It turns it is very easy to mount an inadvertent DOS attack on Zope that kills it. In my case I had a javascript in a periodic timer event handler (with a fairly short period) that did a page reload and under certain circumstances didn't clear the timer causing a timer event driven endless loop. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zope unresponsive
On 2/24/07, Paul Williams [EMAIL PROTECTED] wrote: I have posted this several times, but have not until now been able to get DeadlockDebugger installed. I see several people have had this problem, but no-one has posted a solution. I don't know if that can be the case, but is there a firewall between your Zope and your ZEO? I remember some discussions about firewalls between Zope and ZEO which could cause problems... Regards Marco -- Marco Bizzarri http://iliveinpisa.blogspot.com/ ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] zope unresponsive
Andy Altepeter wrote at 2006-4-24 14:26 -0500: ... I'm experiencing hanging issues with my Zope-2.8.6+zeo setup/ RHEL 4. The hanging isn't categorized by 100% cpu usage. Actually, I had the same issues using 2.8.5, but I've upgraded since then. Here's the situation: I have one zeo client connected to a zeo server on the same box. Apache sits in front, using RewriteRules to request data from zope. After some time (could be 2 minutes or an hour), the zeo client stops responding. Apparently this is called a deadlock or a spinning zope. I know this behaviour from a Python bug triggered by Linux threading peculiarity: In the case of this bug, the main thread is killed by a deadly signal but all other threads remain alife. Therefore, neither the zdaemon nor the clients recognized Zope's death (the zdaemon may recognize it but it cannot restart Zope as the sockets are still in use). A recognizing feature of this bug is that the remaining threads need to be killed with kill -9. This bug is fixed in newest Python versions of the Python 2.3 and Python 2.4 series. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] zope unresponsive
Hey All, I'm experiencing hanging issues with my Zope-2.8.6+zeo setup/ RHEL 4. The hanging isn't categorized by 100% cpu usage. Actually, I had the same issues using 2.8.5, but I've upgraded since then. Here's the situation: I have one zeo client connected to a zeo server on the same box. Apache sits in front, using RewriteRules to request data from zope. After some time (could be 2 minutes or an hour), the zeo client stops responding. Apparently this is called a deadlock or a spinning zope. I've tried using gdb to attach to the zeo client pid, and use the recipe http://zopelabs.com/cookbook/1073504990 to print a traceback, but the call always aborted with SIGABORT. I've captured all of the requests sent to zope during an uptime window (via Z2.log), and using wget to replay the requests. I've also pulled from apache's rewrite log all requests proxied to zope, thinking the Z2.log only writes finished requests. I setup another zeo client (on the same box, different port) and used wget to replay these captures as well. Just running these captures does not cause zope to hang. In fact, I have not been able to cause zope to hang by replaying. There doesn't seem to be any one url or sequence of urls that cause zope to hang. I've tried reinstalling the zope instance, but that didn't help. I've tried using requestprofiler.py to inspect the trace.log. This shows a high number of hangs, but not on a url that actually triggers a spinning zope. Basically, that's where I'm stuck. Is there anything else I can try? Am I missing something? Thanks for the help, Andy ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
