Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On 22.06.09 17:37, Jim Fulton wrote: > > On Jun 22, 2009, at 11:31 AM, Alan Runyan wrote: > >> On Mon, Jun 22, 2009 at 10:06 AM, Andreas Jung wrote: >>> On 22.06.09 16:56, Pedro Ferreira wrote: Hello all, We occasionally get some messages like this in the logs: 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked waiting for storage. Clients waiting: 1. But today we eventually got to: 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked waiting for storage. Clients waiting: 204. Which lasted some minutes. During that time, our system seemed to hang, even if it was possible to connect to the db through the command line. It eventually went back to normal by itself. Any clue on what might have caused this? >>> The reason is likely a long running transaction. Since the storage >>> server >>> is single-threaded, a long running transaction will block pending >>> requests. >> >> Question: >> Will the ZEO server respond to LOAD requests from ZEO clients while >> WRITEs are pending? > > > Yes, however, if the last phase of 2-phase commit takes long enough, > clients will become blocked altogether. This is because vote requests > get blocked and eventually, all of the client threads are blocked voting. I think Dieter's patch is related to this issue: https://bugs.launchpad.net/zodb/+bug/373625 Andreas begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On Jun 22, 2009, at 11:31 AM, Alan Runyan wrote: > On Mon, Jun 22, 2009 at 10:06 AM, Andreas Jung wrote: >> On 22.06.09 16:56, Pedro Ferreira wrote: >>> Hello all, >>> We occasionally get some messages like this in the logs: >>> >>> 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction >>> blocked >>> waiting for storage. Clients waiting: 1. >>> >>> But today we eventually got to: >>> >>> 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked >>> waiting for storage. Clients waiting: 204. >>> >>> Which lasted some minutes. During that time, our system seemed to >>> hang, >>> even if it was possible to connect to the db through the command >>> line. >>> It eventually went back to normal by itself. >>> Any clue on what might have caused this? >> The reason is likely a long running transaction. Since the storage >> server >> is single-threaded, a long running transaction will block pending >> requests. > > Question: > Will the ZEO server respond to LOAD requests from ZEO clients while > WRITEs are pending? Yes, however, if the last phase of 2-phase commit takes long enough, clients will become blocked altogether. This is because vote requests get blocked and eventually, all of the client threads are blocked voting. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On Mon, Jun 22, 2009 at 10:06 AM, Andreas Jung wrote: > On 22.06.09 16:56, Pedro Ferreira wrote: >> Hello all, >> We occasionally get some messages like this in the logs: >> >> 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked >> waiting for storage. Clients waiting: 1. >> >> But today we eventually got to: >> >> 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked >> waiting for storage. Clients waiting: 204. >> >> Which lasted some minutes. During that time, our system seemed to hang, >> even if it was possible to connect to the db through the command line. >> It eventually went back to normal by itself. >> Any clue on what might have caused this? > The reason is likely a long running transaction. Since the storage server > is single-threaded, a long running transaction will block pending requests. Question: Will the ZEO server respond to LOAD requests from ZEO clients while WRITEs are pending? ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On Jun 22, 2009, at 11:06 AM, Andreas Jung wrote: > On 22.06.09 16:56, Pedro Ferreira wrote: >> Hello all, >> We occasionally get some messages like this in the logs: >> >> 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked >> waiting for storage. Clients waiting: 1. >> >> But today we eventually got to: >> >> 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked >> waiting for storage. Clients waiting: 204. >> >> Which lasted some minutes. During that time, our system seemed to >> hang, >> even if it was possible to connect to the db through the command >> line. >> It eventually went back to normal by itself. >> Any clue on what might have caused this? > The reason is likely a long running transaction. Since the storage > server > is single-threaded, a long running transaction will block pending > requests. No, the storage server is only blocked in the very last stage of a two- phase commit, not for the entire duration of the transaction. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On 22.06.09 16:56, Pedro Ferreira wrote: > Hello all, > We occasionally get some messages like this in the logs: > > 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked > waiting for storage. Clients waiting: 1. > > But today we eventually got to: > > 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked > waiting for storage. Clients waiting: 204. > > Which lasted some minutes. During that time, our system seemed to hang, > even if it was possible to connect to the db through the command line. > It eventually went back to normal by itself. > Any clue on what might have caused this? The reason is likely a long running transaction. Since the storage server is single-threaded, a long running transaction will block pending requests. -aj begin:vcard fn:Andreas Jung n:Jung;Andreas org:ZOPYX Ltd. & Co. KG adr;quoted-printable:;;Charlottenstr. 37/1;T=C3=BCbingen;;72070;Germany email;internet:i...@zopyx.com title:CEO tel;work:+49-7071-793376 tel;fax:+49-7071-7936840 tel;home:+49-7071-793257 x-mozilla-html:FALSE url:www.zopyx.com version:2.1 end:vcard ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
Re: [ZODB-Dev] "Transaction blocked waiting for storage"
On Jun 22, 2009, at 10:56 AM, Pedro Ferreira wrote: > Hello all, > We occasionally get some messages like this in the logs: > > 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked > waiting for storage. Clients waiting: 1. This is normal. > But today we eventually got to: > > 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked > waiting for storage. Clients waiting: 204. This isn't. :) > Which lasted some minutes. During that time, our system seemed to > hang, > even if it was possible to connect to the db through the command line. > It eventually went back to normal by itself. > Any clue on what might have caused this? One of your applications took a very long time in the last phase of 2- phase commit (between a tpc_vote and a tpc_finish call). You avoid this in the future, you might want to set the transaction- timeout in your ZEO server: The maximum amount of time to wait for a transaction to commit after acquiring the storage lock, specified in seconds. If the transaction takes too long, the client connection will be closed and the transaction aborted. 30 seconds should normally be more than long enough. Jim -- Jim Fulton Zope Corporation ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
[ZODB-Dev] "Transaction blocked waiting for storage"
Hello all, We occasionally get some messages like this in the logs: 2009-06-22T14:12:42 (30651/137.138.128.213:35787) Transaction blocked waiting for storage. Clients waiting: 1. But today we eventually got to: 2009-06-22T16:00:11 (30651/137.138.4.153:53416) Transaction blocked waiting for storage. Clients waiting: 204. Which lasted some minutes. During that time, our system seemed to hang, even if it was possible to connect to the db through the command line. It eventually went back to normal by itself. Any clue on what might have caused this? Thanks in advance, Pedro -- José Pedro Ferreira (Software Developer, Indico Project) IT-UDS-AVC CERN Geneva, Switzerland Office 513-R-042 Tel. +41 22 76 77159 ___ For more information about ZODB, see the ZODB Wiki: http://www.zope.org/Wikis/ZODB/ ZODB-Dev mailing list - ZODB-Dev@zope.org http://mail.zope.org/mailman/listinfo/zodb-dev
