[Zope] Zeo Client Hanging Unresponsive
I have posted this several times, but have not until now been able to get DeadlockDebugger installed. zope 2.9.5 + zeo pythonm2.4.3 Red Hat RHEL 4 Plone 2.5.1 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. In the event log with tracing on we get Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 There are hundreds to thousands of these until the server is restarted. In the zeo log we get Error caught in asyncor asyncore.py error:(110,'Connection timed out') We have been trying to track this down and have had no luck. Does anyone have any suggestions? Below is our deadlock debugger output Threads traceback dump at 2007-02-23 15:26:50 Thread -1269564496 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training): File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC ore/ZServerPublisher.py", line 23, in __init__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 395, in publish_module File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 196, in publish_module_standard File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat chStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 115, in publish File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m apply.py", line 88, in mapply File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 41, in call_object File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc ripts/Bindings.py", line 311, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc ripts/Bindings.py", line 348, in _bindAndExec File "/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py", line 195, in _exec result = self.pt_render(extra_context=bound_names) File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 38, in FSPT_pt_render result = FSPageTemplate.inheritedAttribute('pt_render')( File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 92, in PT_pt_render tal=not source, strictinsert=0)() File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 238, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 749, in do_useMacro File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 457, in do_optTag_tal File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 442, in do_optTag File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 437, in no_tag File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 749, in do_useMacro File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 507, in do_setLocal_tal File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/TALES.py", line 221, in evaluate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 185, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 180, in _eval File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 85, in render File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 66, in globalize self._initializeData(options=options) File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 147, in _initializeData self._data['language'] = self.request.get('language', None) or \ File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py", line 58, in generatedAccessor return schema[name].get(self, **kw) File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 802, in get value = ObjectField.get(self, instance, **kwargs) File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py"
Re: [Zope] Zeo Client hanging
I see references to asyncore.py near the end of the thread listing when using DeadLockDebugger and seeing a stuck thread process. The tread eventually clears out but sometimes an impatient user reloads, reloads, reloads...and then all 4 threads are stuck on exact same thing. Then you can see the same URL jump to the other Zope/ZEO client and same thing happens there. Another few reloads and one user has effectively crashed the siteuntil the threads clear themselves out. Usually is a catalog search for a news article or something spanning our 6 years of online news articles searching on the word 'bob' or something like that. +1 for DeadlockDebugger... love it. Allen Paul Winkler wrote: On Wed, Oct 18, 2006 at 07:32:38AM +0200, Maciej Wisniowski wrote: zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. Install DeadlockDebugger and you'll be able to see what is the state of your threads when Zope becomes unresponsive. In the event log with tracing on we get Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 There are hundreds to thousands of these until the server is restarted. In the zeo log we get Error caught in asyncore asyncore.py error:(110,'Connection timed out') I'm not sure what these errors are. Maybe firewall between zeo server and zeo client closes the connections or something like that? That was my guess too. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/zope-list/2916870 ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zeo Client hanging
On Wed, Oct 18, 2006 at 07:32:38AM +0200, Maciej Wisniowski wrote: > > > > >zope 2.8.5 + zeo > >python 2.3.4 (red hat distribution) > >Red Hat RHEL 4 > >Plone 2.1.2 > > > >Our zeo clients hang intermittently. We have no way of reproducing the > >problem, but it occurs daily. The client hangs and a restart seems to > >fix the > >problem. > Install DeadlockDebugger and you'll be able to see what is > the state of your threads when Zope becomes unresponsive. > > >In the event log with tracing on we get > > > >Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 > > > >There are hundreds to thousands of these until the server is restarted. > > > >In the zeo log we get > > > >Error caught in asyncore asyncore.py > > > >error:(110,'Connection timed out') > I'm not sure what these errors are. Maybe firewall between zeo > server and zeo client closes the connections or something like that? That was my guess too. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/zope-list/2916870 -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zeo Client hanging
zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. Install DeadlockDebugger and you'll be able to see what is the state of your threads when Zope becomes unresponsive. In the event log with tracing on we get Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 There are hundreds to thousands of these until the server is restarted. In the zeo log we get Error caught in asyncore asyncore.py error:(110,'Connection timed out') I'm not sure what these errors are. Maybe firewall between zeo server and zeo client closes the connections or something like that? -- Maciej Wisniowski ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zeo Client hanging
Hello,We have been having problem with our production servers. We currently have:zope 2.8.5 + zeopython 2.3.4 (red hat distribution)Red Hat RHEL 4Plone 2.1.2Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem.In the event log with tracing on we getTrace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0There are hundreds to thousands of these until the server is restarted.In the zeo log we getError caught in asyncore asyncore.pyerror:(110,'Connection timed out')We have been trying to track this down and have had no luck. Does anyone have any suggestions?Thanks,Paul ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )