Ok, here is what we have. I did a netstat on both machines, client and server. The client sees and established connection and the server does not. In the server log there is a disconnect. As far as hardware between them, there is a switch (dell powerconnect 6024). Web Server Directors might get hold of it but there are no hops on traceroute. Traceroute only shows the client machine and the server machine.

So the client is just continuously polling the connection but getting nothing back.


What we are thinking about doing is changing the code in zrpc/connection.py to close the connection in wait (line 638 zope version 2.9.5) if the wait time gets too large or the poll has happened too many times.

We are great at plone development, but have very little backend zope development. Would someone please advise me as to whether this is going to cause more problems?

Thanks,
Paul Williams

Paul Williams wrote:
I have posted this several times, but have not until now been able to get DeadlockDebugger installed. I see several people have had this problem, but no-one has posted a solution.

zope 2.9.5 + zeo

pythonm2.4.3

Red Hat RHEL 4

Plone 2.5.1

Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the
problem.

In the event log with tracing on we get

Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0

There are hundreds to thousands of these until the server is restarted.

In the zeo log we get

Error caught in asyncor asyncore.py

error:(110,'Connection timed out')

We have been trying to track this down and have had no luck. Does anyone have
any suggestions?  Below is our deadlock debugger output

Threads traceback dump at 2007-02-23 15:26:50

Thread -1269564496 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training):

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard

File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 115, in publish

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py", line 88, in mapply

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 41, in call_object

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py", line 311, in __call__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py", line 348, in _bindAndExec

File "/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py", line 195, in _exec

    result = self.pt_render(extra_context=bound_names)

File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 38, in FSPT_pt_render

    result = FSPageTemplate.inheritedAttribute('pt_render')(

File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 92, in PT_pt_render

    tal=not source, strictinsert=0)()

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 238, in __call__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 457, in do_optTag_tal

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 442, in do_optTag

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 437, in no_tag

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 507, in do_setLocal_tal

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/TALES.py", line 221, in evaluate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 185, in __call__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 180, in _eval

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 85, in render

File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 66, in globalize

    self._initializeData(options=options)

File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 147, in _initializeData

    self._data['language'] =  self.request.get('language', None) or \

File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py", line 58, in generatedAccessor

    return schema[name].get(self, **kw)

File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 802, in get

    value = ObjectField.get(self, instance, **kwargs)

File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 671, in get

    return self.getStorage(instance).get(self.getName(), instance, **kwargs)

File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Storage/__init__.py", line 175, in get

    value = base._md[name]

File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/UserDict.py", line 17, in __getitem__

    def __getitem__(self, key): return self.data[key]

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx

Thread -1290544208 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content//nav):

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard

File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 106, in publish

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py", line 366, in traverse

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx

Thread -1246884944 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/carrier.jpg):

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard

File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 115, in publish

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py", line 88, in mapply

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 41, in call_object

File "/apps1/zope2.9.5/navo_instance/Products/ATContentTypes/content/base.py", line 414, in index_html

    if data:

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 769, in loadEx

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ServerStub.py", line 192, in loadEx

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py", line 531, in call

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py", line 638, in wait

File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/asyncore.py", line 122, in poll

    r, w, e = select.select(r, w, e, timeout)

Thread -1280054352 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/):

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard

File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 106, in publish

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py", line 366, in traverse

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load

File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx

End of dump

Thank you,

Paul Williams


------------------------------------------------------------------------

_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )

_______________________________________________
Zope maillist  -  Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )

Reply via email to