Just went throught that exercise with Andy and installed a patch to
MySQLDA that effectively ignores the 'release unlocked lock' problem that
has been plaguing us. I shoulda guessed that is the first place to look.
I'll get and install the DeadlockDebugger forthwith.
On Mon, 28 Nov 2005, Florent Guillaume wrote:
> Dennis Allison wrote:
> > We have two recent instances in our production sites where Zope suddenly
> > stops responding. It is not a new problem, but we've now been confronted
> > with two clean examples and nothing to blame them on. The problem appears
> > to be independent of load as both incidents were on lightly loaded
> > machines.
> > A check of the logs (Linux and Zope) shows nothing obviously amiss except
> > that the trace log (the old -M log) shows a sudden increase in active
> > requests from the typical 0 or 1 to 1300 or more. In this context an
> > "active request' is total number of requests pending at the end of this
> > request and is computed by post-processing. We front-end Zope with pound
> > and make heavy use of MySQL. Both show a plethora of incomplete
> > transactions.
> > Examination of the raw trace log shows that Zope is continuing to accept
> > requests, but nothing getting done. The raw log date-stamps four internal
> > states for each transaction. The states are Begin (B), Input (I),
> > action (A), and End (E). Inputs are gathered between B and I, outputs is
> > made between A and E. The raw log shows B and I transactions, but
> > apparently no processing is completing. I suspect that nothing is getting
> > scheduled.
> > I am at a loss as to where to begin to track this one down. The failure
> > is spontaneous and apparently not triggered by any readily distinguishable
> > inputs or pattern of inputs. The behavior smells a bit of resource limits
> > or process synchronization problems, but there is not real evidence for
> > either being the root cause. I am not sure what monitoring I should be
> > doing to help locate the source of the problem.
> > Has anyone seen seen a similar problem? Any advice as to how to proceed?
> Threads are hanging. You should install my DeadlockDebugger and track
> down where the hung threads are blocked at.
> From the description I'd wager that you'll find your threads stuck in a
> corner of the MySQL DA. In which case you'd have to find why it
> deadlocks and find a fix.
Zope maillist - Zope@zope.org
** No cross posts or HTML encoding! **
(Related lists -