RE: [ZODB-Dev] zeo client patch in connection.py
I did use the deadlockdebugger. There were three threads doing folder listing and the last one was in the wait method doing a continuous loop listening for the server. The server was disconnected. Somehow the client just never knew. -Original Message- From: Christian Theune [mailto:[EMAIL PROTECTED] Sent: Thursday, March 22, 2007 2:45 AM To: Paul Williams Cc: zodb-dev@zope.org Subject: RE: [ZODB-Dev] zeo client patch in connection.py Hi, I haven't had a look at your patch earlier. With your error description and with the patch together, it makes sense to me. You patched the "wait" method which is described as: def wait(self, msgid): """Invoke asyncore mainloop and wait for reply.""" This method is used to wait for any reply to function invoked on the server. However, you limited the response time to a few seconds. That's why long running tasks that do not respond within this time frame will disconnect the client. Your client lockup problem doesn't sound like something ZODB would do on it's own. I'd ponder checking whether your application code does something weird. Try going back to the original state and, when getting lockups, throw in DeadLockDebugger (remember to install first and test that it shows you the processes, before getting a lockup) and find out in what part of the code the threads are hanging. Christian Am Mittwoch, den 21.03.2007, 13:06 -0500 schrieb Paul Williams: > Since we implemented this, any task that takes more than a few seconds (ie. > pack the database) throws a clientdisconnectederror. I am hoping someone > out there can shed some light on what might be happening. > > -Original Message- > From: Christian Theune [mailto:[EMAIL PROTECTED] > Sent: Wednesday, March 07, 2007 9:43 AM > To: Paul Williams > Cc: zodb-dev@zope.org > Subject: Re: [ZODB-Dev] zeo client patch in connection.py > > Hi, > > could you send this again as a patch produced using `diff` against your > original sources OR tell us which version of Zope/ZODB you used as the > starting point for this patch. > > Thanks, > Christian > > Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams: > > Hi Everyone, > > > > We were experience problems with our zeo client setup on redhat rhel4. > > The client would just quit responding. No memory or cpu increase was > > associated with this. The client would remain hung until it was > > restarted. > > > > We looked on the client using Netstat and the status was ESTABLISHED > > with the zeo server. On the zeo server the netstat said LISTENING. > > > > When running the deadlockdebugger, one thread was in asnycore wait. > > The others were normal actions such as folder listing or folder > > contents. > > > > We implemented a couple of lines of code on line 641 of connection.py > > in the ZEO/zrpc packages > > > > We added and else clause to call self.close() if delay is over one > > second. We found that one second wasn't quite enough and moved it to > > 5 seconds. > > > > Now we find out that this drastically improved our performance. The > > servers are now < 1 second per page load. Before, they could be 5 > > seconds or more, if they loaded at all. Also, our servers used to > > crash several times a day and they now haven't crashed in almost a > > week. > > > > I just wanted to put this out there and see if anyone has any comments > > at all. I need to get a more permanent solution than this, but it is > > what we have for now. > > > > System Configuration > > Zope 2.9.5 > > Plone 2.5.1 > > Python 2.4.3 > > Redhat Rhel4 > > > > Communications between our zeo clients and zeo server only route > > through a switch. > > > > > > Thank you for any help, > > Paul Williams > > > > > > ___ > > For more information about ZODB, see the ZODB Wiki: > > http://www.zope.org/Wikis/ZODB/ > > > > ZODB-Dev mailing list - ZODB-Dev@zope.org > > http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zeo client patch in connection.py
Hi, I haven't had a look at your patch earlier. With your error description and with the patch together, it makes sense to me. You patched the "wait" method which is described as: def wait(self, msgid): """Invoke asyncore mainloop and wait for reply.""" This method is used to wait for any reply to function invoked on the server. However, you limited the response time to a few seconds. That's why long running tasks that do not respond within this time frame will disconnect the client. Your client lockup problem doesn't sound like something ZODB would do on it's own. I'd ponder checking whether your application code does something weird. Try going back to the original state and, when getting lockups, throw in DeadLockDebugger (remember to install first and test that it shows you the processes, before getting a lockup) and find out in what part of the code the threads are hanging. Christian Am Mittwoch, den 21.03.2007, 13:06 -0500 schrieb Paul Williams: > Since we implemented this, any task that takes more than a few seconds (ie. > pack the database) throws a clientdisconnectederror. I am hoping someone > out there can shed some light on what might be happening. > > -Original Message- > From: Christian Theune [mailto:[EMAIL PROTECTED] > Sent: Wednesday, March 07, 2007 9:43 AM > To: Paul Williams > Cc: zodb-dev@zope.org > Subject: Re: [ZODB-Dev] zeo client patch in connection.py > > Hi, > > could you send this again as a patch produced using `diff` against your > original sources OR tell us which version of Zope/ZODB you used as the > starting point for this patch. > > Thanks, > Christian > > Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams: > > Hi Everyone, > > > > We were experience problems with our zeo client setup on redhat rhel4. > > The client would just quit responding. No memory or cpu increase was > > associated with this. The client would remain hung until it was > > restarted. > > > > We looked on the client using Netstat and the status was ESTABLISHED > > with the zeo server. On the zeo server the netstat said LISTENING. > > > > When running the deadlockdebugger, one thread was in asnycore wait. > > The others were normal actions such as folder listing or folder > > contents. > > > > We implemented a couple of lines of code on line 641 of connection.py > > in the ZEO/zrpc packages > > > > We added and else clause to call self.close() if delay is over one > > second. We found that one second wasn't quite enough and moved it to > > 5 seconds. > > > > Now we find out that this drastically improved our performance. The > > servers are now < 1 second per page load. Before, they could be 5 > > seconds or more, if they loaded at all. Also, our servers used to > > crash several times a day and they now haven't crashed in almost a > > week. > > > > I just wanted to put this out there and see if anyone has any comments > > at all. I need to get a more permanent solution than this, but it is > > what we have for now. > > > > System Configuration > > Zope 2.9.5 > > Plone 2.5.1 > > Python 2.4.3 > > Redhat Rhel4 > > > > Communications between our zeo clients and zeo server only route > > through a switch. > > > > > > Thank you for any help, > > Paul Williams > > > > > > ___ > > For more information about ZODB, see the ZODB Wiki: > > http://www.zope.org/Wikis/ZODB/ > > > > ZODB-Dev mailing list - ZODB-Dev@zope.org > > http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zeo client patch in connection.py
Since we implemented this, any task that takes more than a few seconds (ie. pack the database) throws a clientdisconnectederror. I am hoping someone out there can shed some light on what might be happening. -Original Message- From: Christian Theune [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 9:43 AM To: Paul Williams Cc: zodb-dev@zope.org Subject: Re: [ZODB-Dev] zeo client patch in connection.py Hi, could you send this again as a patch produced using `diff` against your original sources OR tell us which version of Zope/ZODB you used as the starting point for this patch. Thanks, Christian Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams: > Hi Everyone, > > We were experience problems with our zeo client setup on redhat rhel4. > The client would just quit responding. No memory or cpu increase was > associated with this. The client would remain hung until it was > restarted. > > We looked on the client using Netstat and the status was ESTABLISHED > with the zeo server. On the zeo server the netstat said LISTENING. > > When running the deadlockdebugger, one thread was in asnycore wait. > The others were normal actions such as folder listing or folder > contents. > > We implemented a couple of lines of code on line 641 of connection.py > in the ZEO/zrpc packages > > We added and else clause to call self.close() if delay is over one > second. We found that one second wasn't quite enough and moved it to > 5 seconds. > > Now we find out that this drastically improved our performance. The > servers are now < 1 second per page load. Before, they could be 5 > seconds or more, if they loaded at all. Also, our servers used to > crash several times a day and they now haven't crashed in almost a > week. > > I just wanted to put this out there and see if anyone has any comments > at all. I need to get a more permanent solution than this, but it is > what we have for now. > > System Configuration > Zope 2.9.5 > Plone 2.5.1 > Python 2.4.3 > Redhat Rhel4 > > Communications between our zeo clients and zeo server only route > through a switch. > > > Thank you for any help, > Paul Williams > > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
RE: [ZODB-Dev] zeo client patch in connection.py
Here is a patch. The version is 3.6.2 and was distributed with Zope 2.9.5. Thank you, Paul Williams -Original Message- From: Christian Theune [mailto:[EMAIL PROTECTED] Sent: Wednesday, March 07, 2007 9:43 AM To: Paul Williams Cc: zodb-dev@zope.org Subject: Re: [ZODB-Dev] zeo client patch in connection.py Hi, could you send this again as a patch produced using `diff` against your original sources OR tell us which version of Zope/ZODB you used as the starting point for this patch. Thanks, Christian Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams: > Hi Everyone, > > We were experience problems with our zeo client setup on redhat rhel4. > The client would just quit responding. No memory or cpu increase was > associated with this. The client would remain hung until it was > restarted. > > We looked on the client using Netstat and the status was ESTABLISHED > with the zeo server. On the zeo server the netstat said LISTENING. > > When running the deadlockdebugger, one thread was in asnycore wait. > The others were normal actions such as folder listing or folder > contents. > > We implemented a couple of lines of code on line 641 of connection.py > in the ZEO/zrpc packages > > We added and else clause to call self.close() if delay is over one > second. We found that one second wasn't quite enough and moved it to > 5 seconds. > > Now we find out that this drastically improved our performance. The > servers are now < 1 second per page load. Before, they could be 5 > seconds or more, if they loaded at all. Also, our servers used to > crash several times a day and they now haven't crashed in almost a > week. > > I just wanted to put this out there and see if anyone has any comments > at all. I need to get a more permanent solution than this, but it is > what we have for now. > > System Configuration > Zope 2.9.5 > Plone 2.5.1 > Python 2.4.3 > Redhat Rhel4 > > Communications between our zeo clients and zeo server only route > through a switch. > > > Thank you for any help, > Paul Williams > > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development connection.py.patch Description: connection.py.patch ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] zeo client patch in connection.py
Hi, could you send this again as a patch produced using `diff` against your original sources OR tell us which version of Zope/ZODB you used as the starting point for this patch. Thanks, Christian Am Mittwoch, den 07.03.2007, 08:52 -0600 schrieb Paul Williams: > Hi Everyone, > > We were experience problems with our zeo client setup on redhat rhel4. > The client would just quit responding. No memory or cpu increase was > associated with this. The client would remain hung until it was > restarted. > > We looked on the client using Netstat and the status was ESTABLISHED > with the zeo server. On the zeo server the netstat said LISTENING. > > When running the deadlockdebugger, one thread was in asnycore wait. > The others were normal actions such as folder listing or folder > contents. > > We implemented a couple of lines of code on line 641 of connection.py > in the ZEO/zrpc packages > > We added and else clause to call self.close() if delay is over one > second. We found that one second wasn't quite enough and moved it to > 5 seconds. > > Now we find out that this drastically improved our performance. The > servers are now < 1 second per page load. Before, they could be 5 > seconds or more, if they loaded at all. Also, our servers used to > crash several times a day and they now haven't crashed in almost a > week. > > I just wanted to put this out there and see if anyone has any comments > at all. I need to get a more permanent solution than this, but it is > what we have for now. > > System Configuration > Zope 2.9.5 > Plone 2.5.1 > Python 2.4.3 > Redhat Rhel4 > > Communications between our zeo clients and zeo server only route > through a switch. > > > Thank you for any help, > Paul Williams > > > ___ > For more information about ZODB, see the ZODB Wiki: > http://www.zope.org/Wikis/ZODB/ > > ZODB-Dev mailing list - ZODB-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zodb-dev -- gocept gmbh & co. kg - forsterstraße 29 - 06112 halle/saale - germany www.gocept.com - [EMAIL PROTECTED] - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development signature.asc Description: Dies ist ein digital signierter Nachrichtenteil ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
