Re: [Zope] Zope 2.8.4 strange behavior
We do see occasional instances where things slow down, a backlog builds, and then it clears. But no disconnects. On Sun, 27 Nov 2005, Paul Winkler wrote: > On Sun, Nov 27, 2005 at 08:08:20PM -0800, Dennis Allison wrote: > > Thanks Paul. > > > > I checked and did not find any "Client disconnected" entries in the > > event log so this may not be the cause of what I am seeing. > > OK. If you were having my problem you'd see them for sure > when the problem stops (for me it always fixes itself "eventually"). > > > I will > > bump the log level to ALL (which I think includes BLATHER) > > Yep. > > > when I restart things tonight. > > > > We use a remote ZEO but at the moment Zope and ZEO share the same machine. > > You're reasonably unlikely to have my problem then :-) > > -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
We had been plagued with a threading error which manifests itself
as an exception reporting "release unreleased lock". At Andy's
suggestion I added code to catch the exception in the appropriate
place in ZMySQLDA --
Here's the code from db.py ---
The code probably should be specific to the unreleased lock exception
since, presumably, the code could throw other exceptions. On the whole,
I think this is a ban-aid solution.
def _abort(self, *ignored):
try:
if self._mysql_lock:
self.db.query("SELECT RELEASE_LOCK('%s')" %
self._mysql_lock)
self.db.store_result()
if self._transactions:
self.db.query("ROLLBACK")
self.db.store_result()
else:
LOG('ZMySQLDA', ERROR, "aborting when non-transactional")
finally:
# just ignore the exception for "release unlocked lock"
try:
self._tlock.release()
except:
pass
On Sun, 27 Nov 2005, Chris McDonough wrote:
> > Chris,
> >
> > A helpful suggestion. The commit errors I've been seeing have to
> > do with the intereaction of the ZODB, MySQL, session variables, and
> > conflicts.
>
> So the patch that Andy sent over is a fix that prevents the mysql
> adapter from raising an error when a conflict exception occurs? Do
> you know if that error only happens after a conflict exception occurs
> or can it happen without the presence of conflict exceptions? (I'm
> curious because I also use the adapter and I'd like to know what the
> patch fixes).
>
>
> > These particular problems do not appear to be related to the ZODB/
> > session
> > variable/conflict issues, but I cannot completely exclude them
> > since leaks
> > easily cause failures far away from where the fault lies.
>
> I'm afraid I can't parse that sentence fully. But I'll try to
> interpret as best possible. ;-)
>
> I think I've said this before but it in case not... the use of
> sessions is only one place where conflict errors can be generated.
> Conflict errors are "normal" in any system that causes writes to a
> ZODB database. If your application does any writes to a ZODB
> database at all (besides the writes that occur from use of the
> sessioning machinery), and the mysql adapter wasn't tolerant of
> conflict errors, you'd be getting the same result, they'd just
> probably happen further apart.
>
> That said, I certainly am interested in making fewer conflicts errors
> go unresolved when the sessioning machinery is used. If you were to
> tell that you've taken the steps I've already suggested about
> reducing the potential for conflicts during session usage (use 2.8
> with mvcc, turn external housekeeping on, bump up the resolution
> time, local zodb db for sessions), and you observed that you're still
> having "too many" conflicts, I'd try to take some action, although to
> do so I might still need to request your help in providing data about
> your conflict rates.
>
> > It certainly is a worthwhile thing to check and monitor.
>
> What is?
>
> Thanks,
>
> - C
>
--
___
Zope maillist - Zope@zope.org
http://mail.zope.org/mailman/listinfo/zope
** No cross posts or HTML encoding! **
(Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
On Sun, Nov 27, 2005 at 08:08:20PM -0800, Dennis Allison wrote: > Thanks Paul. > > I checked and did not find any "Client disconnected" entries in the > event log so this may not be the cause of what I am seeing. OK. If you were having my problem you'd see them for sure when the problem stops (for me it always fixes itself "eventually"). > I will > bump the log level to ALL (which I think includes BLATHER) Yep. > when I restart things tonight. > > We use a remote ZEO but at the moment Zope and ZEO share the same machine. You're reasonably unlikely to have my problem then :-) -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
Thanks Paul. I checked and did not find any "Client disconnected" entries in the event log so this may not be the cause of what I am seeing. I will bump the log level to ALL (which I think includes BLATHER) when I restart things tonight. We use a remote ZEO but at the moment Zope and ZEO share the same machine. On Sun, 27 Nov 2005, Paul Winkler wrote: > On Sun, Nov 27, 2005 at 05:13:21PM -0800, Dennis Allison wrote: > > Examination of the raw trace log shows that Zope is continuing to accept > > requests, but nothing getting done. The raw log date-stamps four internal > > states for each transaction. The states are Begin (B), Input (I), > > action (A), and End (E). Inputs are gathered between B and I, outputs is > > made between A and E. The raw log shows B and I transactions, but > > apparently no processing is completing. I suspect that nothing is getting > > scheduled. > > I've seen the same symptoms a number of times recently with zope 2.7.x. > In our case, it seems to be related to ZEO. Zope seems to have lost > its connection to ZEO but doesn't realize it somehow. My theory is > that the symptom starts when all worker threads are waiting for objects > that aren't in the ZEO client cache, so they're all waiting on ZEO > requests. Meanwhile, requests keep piling up in the queue. > These are all the B and I lines you noticed. > > Eventually (10 minutes or so) one of these ZEO requests times out, and > Zope then realizes it's lost its ZEO connection and successfully reconnects. > This shows up in the Zope event log as a series of ZEO "Client > disconnected" errors, and immediately thereafter we see a successful > reconnect in the ZEO server's log. > > I have no idea what is causing the loss of connection in the first > place, or why it takes Zope so long to realize there's a problem. > Nothing in the logs so far gives me any clue. I've bumped up our log > levels to BLATHER and hopefully next time this happens I'll get some > more to work with and see if there's anythign I can do about it. > > It may be network issues between our Zope and Zeo boxes, which > are on different subnets with a firewall between them, over which > I have no control. > > -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope 2.8.4 strange behavior
I have not seen any of the threading error exceptions--but then the patch catches them in the ZMySQLDA adaptor and punts... On Sun, 27 Nov 2005, Chris McDonough wrote: > Does this mean that you haven't seen the errors since installing > Andy's patch? If not, I'd declare victory and forget about using the > deadlock debugger (unless you want to do it for learning purposes only). > > On Nov 27, 2005, at 8:46 PM, Dennis Allison wrote: > > > > > Just went throught that exercise with Andy and installed a patch to > > MySQLDA that effectively ignores the 'release unlocked lock' > > problem that > > has been plaguing us. I shoulda guessed that is the first place > > to look. > > > > I'll get and install the DeadlockDebugger forthwith. > > > > Thanks. > -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
Chris, A helpful suggestion. The commit errors I've been seeing have to do with the intereaction of the ZODB, MySQL, session variables, and conflicts. So the patch that Andy sent over is a fix that prevents the mysql adapter from raising an error when a conflict exception occurs? Do you know if that error only happens after a conflict exception occurs or can it happen without the presence of conflict exceptions? (I'm curious because I also use the adapter and I'd like to know what the patch fixes). These particular problems do not appear to be related to the ZODB/ session variable/conflict issues, but I cannot completely exclude them since leaks easily cause failures far away from where the fault lies. I'm afraid I can't parse that sentence fully. But I'll try to interpret as best possible. ;-) I think I've said this before but it in case not... the use of sessions is only one place where conflict errors can be generated. Conflict errors are "normal" in any system that causes writes to a ZODB database. If your application does any writes to a ZODB database at all (besides the writes that occur from use of the sessioning machinery), and the mysql adapter wasn't tolerant of conflict errors, you'd be getting the same result, they'd just probably happen further apart. That said, I certainly am interested in making fewer conflicts errors go unresolved when the sessioning machinery is used. If you were to tell that you've taken the steps I've already suggested about reducing the potential for conflicts during session usage (use 2.8 with mvcc, turn external housekeeping on, bump up the resolution time, local zodb db for sessions), and you observed that you're still having "too many" conflicts, I'd try to take some action, although to do so I might still need to request your help in providing data about your conflict rates. It certainly is a worthwhile thing to check and monitor. What is? Thanks, - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
On Sun, Nov 27, 2005 at 05:13:21PM -0800, Dennis Allison wrote: > Examination of the raw trace log shows that Zope is continuing to accept > requests, but nothing getting done. The raw log date-stamps four internal > states for each transaction. The states are Begin (B), Input (I), > action (A), and End (E). Inputs are gathered between B and I, outputs is > made between A and E. The raw log shows B and I transactions, but > apparently no processing is completing. I suspect that nothing is getting > scheduled. I've seen the same symptoms a number of times recently with zope 2.7.x. In our case, it seems to be related to ZEO. Zope seems to have lost its connection to ZEO but doesn't realize it somehow. My theory is that the symptom starts when all worker threads are waiting for objects that aren't in the ZEO client cache, so they're all waiting on ZEO requests. Meanwhile, requests keep piling up in the queue. These are all the B and I lines you noticed. Eventually (10 minutes or so) one of these ZEO requests times out, and Zope then realizes it's lost its ZEO connection and successfully reconnects. This shows up in the Zope event log as a series of ZEO "Client disconnected" errors, and immediately thereafter we see a successful reconnect in the ZEO server's log. I have no idea what is causing the loss of connection in the first place, or why it takes Zope so long to realize there's a problem. Nothing in the logs so far gives me any clue. I've bumped up our log levels to BLATHER and hopefully next time this happens I'll get some more to work with and see if there's anythign I can do about it. It may be network issues between our Zope and Zeo boxes, which are on different subnets with a firewall between them, over which I have no control. -- Paul Winkler http://www.slinkp.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
Chris, A helpful suggestion. The commit errors I've been seeing have to do with the intereaction of the ZODB, MySQL, session variables, and conflicts. These particular problems do not appear to be related to the ZODB/session variable/conflict issues, but I cannot completely exclude them since leaks easily cause failures far away from where the fault lies. It certainly is a worthwhile thing to check and monitor. -d On Sun, 27 Nov 2005, Chris McDonough wrote: > > On Nov 27, 2005, at 8:13 PM, Dennis Allison wrote: > > > here to begin to track this one down. The failure > > is spontaneous and apparently not triggered by any readily > > distinguishable > > inputs or pattern of inputs. The behavior smells a bit of resource > > limits > > or process synchronization problems, but there is not real evidence > > for > > either being the root cause. I am not sure what monitoring I > > should be > > doing to help locate the source of the problem. > > > > Has anyone seen seen a similar problem? Any advice as to how to > > proceed? > > A pool of database connections is kept around to pull from for each > ZODB database you have. By default, there are 7 of them per > database; this number can be changed by the pool-size parameter in > the section in your config file. These connections are > returned to the pool when a connection is closed. This usually > happens when a transaction that is using that connection completes. > > The symptom you're describing has happened to me in the past. Zope > stops serving requests and my CPU is not pegged (it's not hitting the > CPU hard via an infinept loop I've made). I usually end up tracking > it down to the fact that my instance is somehow leaking database > connections. The connection is never closed so it never gets > returned to the pool. This can happen when there is an error during > a transaction commit or abort. > > Since you've previously posted to the list with commit errors of just > this kind, it seems like that would be the logical place to start. > You didn't mention seeing any of these errors, though, so if you > don't see any of those in the event log, I'm not sure where to go > from there other than debugging by binary exclusion which can be > impossible when the failures happen so far apart. :-( > > - C > -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope 2.8.4 strange behavior
Does this mean that you haven't seen the errors since installing Andy's patch? If not, I'd declare victory and forget about using the deadlock debugger (unless you want to do it for learning purposes only). On Nov 27, 2005, at 8:46 PM, Dennis Allison wrote: Just went throught that exercise with Andy and installed a patch to MySQLDA that effectively ignores the 'release unlocked lock' problem that has been plaguing us. I shoulda guessed that is the first place to look. I'll get and install the DeadlockDebugger forthwith. Thanks. ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Zope 2.8.4 strange behavior
On Nov 27, 2005, at 8:13 PM, Dennis Allison wrote: here to begin to track this one down. The failure is spontaneous and apparently not triggered by any readily distinguishable inputs or pattern of inputs. The behavior smells a bit of resource limits or process synchronization problems, but there is not real evidence for either being the root cause. I am not sure what monitoring I should be doing to help locate the source of the problem. Has anyone seen seen a similar problem? Any advice as to how to proceed? A pool of database connections is kept around to pull from for each ZODB database you have. By default, there are 7 of them per database; this number can be changed by the pool-size parameter in the section in your config file. These connections are returned to the pool when a connection is closed. This usually happens when a transaction that is using that connection completes. The symptom you're describing has happened to me in the past. Zope stops serving requests and my CPU is not pegged (it's not hitting the CPU hard via an infinept loop I've made). I usually end up tracking it down to the fact that my instance is somehow leaking database connections. The connection is never closed so it never gets returned to the pool. This can happen when there is an error during a transaction commit or abort. Since you've previously posted to the list with commit errors of just this kind, it seems like that would be the logical place to start. You didn't mention seeing any of these errors, though, so if you don't see any of those in the event log, I'm not sure where to go from there other than debugging by binary exclusion which can be impossible when the failures happen so far apart. :-( - C ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Re: Zope 2.8.4 strange behavior
Just went throught that exercise with Andy and installed a patch to MySQLDA that effectively ignores the 'release unlocked lock' problem that has been plaguing us. I shoulda guessed that is the first place to look. I'll get and install the DeadlockDebugger forthwith. Thanks. On Mon, 28 Nov 2005, Florent Guillaume wrote: > Dennis Allison wrote: > > We have two recent instances in our production sites where Zope suddenly > > stops responding. It is not a new problem, but we've now been confronted > > with two clean examples and nothing to blame them on. The problem appears > > to be independent of load as both incidents were on lightly loaded > > machines. > > > > A check of the logs (Linux and Zope) shows nothing obviously amiss except > > that the trace log (the old -M log) shows a sudden increase in active > > requests from the typical 0 or 1 to 1300 or more. In this context an > > "active request' is total number of requests pending at the end of this > > request and is computed by post-processing. We front-end Zope with pound > > and make heavy use of MySQL. Both show a plethora of incomplete > > transactions. > > > > Examination of the raw trace log shows that Zope is continuing to accept > > requests, but nothing getting done. The raw log date-stamps four internal > > states for each transaction. The states are Begin (B), Input (I), > > action (A), and End (E). Inputs are gathered between B and I, outputs is > > made between A and E. The raw log shows B and I transactions, but > > apparently no processing is completing. I suspect that nothing is getting > > scheduled. > > > > I am at a loss as to where to begin to track this one down. The failure > > is spontaneous and apparently not triggered by any readily distinguishable > > inputs or pattern of inputs. The behavior smells a bit of resource limits > > or process synchronization problems, but there is not real evidence for > > either being the root cause. I am not sure what monitoring I should be > > doing to help locate the source of the problem. > > > > Has anyone seen seen a similar problem? Any advice as to how to proceed? > > Threads are hanging. You should install my DeadlockDebugger and track > down where the hung threads are blocked at. > > From the description I'd wager that you'll find your threads stuck in a > corner of the MySQL DA. In which case you'd have to find why it > deadlocks and find a fix. > > Florent > > -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Re: Zope 2.8.4 strange behavior
Dennis Allison wrote: We have two recent instances in our production sites where Zope suddenly stops responding. It is not a new problem, but we've now been confronted with two clean examples and nothing to blame them on. The problem appears to be independent of load as both incidents were on lightly loaded machines. A check of the logs (Linux and Zope) shows nothing obviously amiss except that the trace log (the old -M log) shows a sudden increase in active requests from the typical 0 or 1 to 1300 or more. In this context an "active request' is total number of requests pending at the end of this request and is computed by post-processing. We front-end Zope with pound and make heavy use of MySQL. Both show a plethora of incomplete transactions. Examination of the raw trace log shows that Zope is continuing to accept requests, but nothing getting done. The raw log date-stamps four internal states for each transaction. The states are Begin (B), Input (I), action (A), and End (E). Inputs are gathered between B and I, outputs is made between A and E. The raw log shows B and I transactions, but apparently no processing is completing. I suspect that nothing is getting scheduled. I am at a loss as to where to begin to track this one down. The failure is spontaneous and apparently not triggered by any readily distinguishable inputs or pattern of inputs. The behavior smells a bit of resource limits or process synchronization problems, but there is not real evidence for either being the root cause. I am not sure what monitoring I should be doing to help locate the source of the problem. Has anyone seen seen a similar problem? Any advice as to how to proceed? Threads are hanging. You should install my DeadlockDebugger and track down where the hung threads are blocked at. From the description I'd wager that you'll find your threads stuck in a corner of the MySQL DA. In which case you'd have to find why it deadlocks and find a fix. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com [EMAIL PROTECTED] ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Zope 2.8.4 strange behavior
Python2.4.2 Zope 2.8.4 ZODB/Zeo 2.3.4 Mysql 4.0 Dual Athalon processors Linux RH7.3 We have two recent instances in our production sites where Zope suddenly stops responding. It is not a new problem, but we've now been confronted with two clean examples and nothing to blame them on. The problem appears to be independent of load as both incidents were on lightly loaded machines. A check of the logs (Linux and Zope) shows nothing obviously amiss except that the trace log (the old -M log) shows a sudden increase in active requests from the typical 0 or 1 to 1300 or more. In this context an "active request' is total number of requests pending at the end of this request and is computed by post-processing. We front-end Zope with pound and make heavy use of MySQL. Both show a plethora of incomplete transactions. Examination of the raw trace log shows that Zope is continuing to accept requests, but nothing getting done. The raw log date-stamps four internal states for each transaction. The states are Begin (B), Input (I), action (A), and End (E). Inputs are gathered between B and I, outputs is made between A and E. The raw log shows B and I transactions, but apparently no processing is completing. I suspect that nothing is getting scheduled. I am at a loss as to where to begin to track this one down. The failure is spontaneous and apparently not triggered by any readily distinguishable inputs or pattern of inputs. The behavior smells a bit of resource limits or process synchronization problems, but there is not real evidence for either being the root cause. I am not sure what monitoring I should be doing to help locate the source of the problem. Has anyone seen seen a similar problem? Any advice as to how to proceed? -- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] xform in zope 2.8
Does anybody know how to monkey patch 2.8.0 so zope can process an xform? I found the xmlFix patch but a fellow zope user pointed out that 2.8.1 breaks the monkey patch because of the zope.app.publication.HTTPPublicationRequestFactory object. Does 2.8.0 have the HTTPPublicationRequestFactory object? I cannot find it but then I also cannot seem to process the xform.Thanks!Tim ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] (no subject)
-- ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
