Re: [Zope] Counting threads
GDB can help you determine where the hanging condition originates. Here's some links: http://plone.org/documentation/how-to/use-gdb-to-get-backtraces-of-hung-spinning-sites http://zopelabs.com/cookbook/1073504990 This is great! I haven't used it yet but it definitely appears to me the right tool for me. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
Peter Bengtsson wrote at 2008-1-22 22:33 +: ... By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma. Use the DeadlockDebugger to find out what your Zope is doing. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
[Zope] Counting threads
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. They probably die when being designated to open an external relational database connection or something dodgy. What are my options for monitoring and debugging what zope's threads are doing? Is there a simple tool to just get a simple readout of how many threads are busy? -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
On Jan 22, 2008, at 16:23 , Peter Bengtsson wrote: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. They probably die when being designated to open an external relational database connection or something dodgy. What are my options for monitoring and debugging what zope's threads are doing? Is there a simple tool to just get a simple readout of how many threads are busy? One simple option is the ZMI Control_Panel Debug screen. Go to / Control_Panel/DebugInfo/manage_main on your instance and scroll to the bottom. It shows current connections to the Zope server from the outside, and how long the connection has been around. You will always at least see one entry, which is the request to fetch that debug screen. If you have connections beyond a few seconds of age they may be stuck. jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
Peter Bengtsson wrote at 2008-1-22 15:23 +: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason. Zope uses a worker thread pool (of size 4, by default), allocated quite early. It does not create further threads. It is therefore very unlikely that it hits a thread limit. -- Dieter ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
On 22/01/2008, Dieter Maurer [EMAIL PROTECTED] wrote: Peter Bengtsson wrote at 2008-1-22 15:23 +: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason. By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma. Zope uses a worker thread pool (of size 4, by default), allocated quite early. It does not create further threads. It is therefore very unlikely that it hits a thread limit. I'm just not certain if it's this. It seems highly related to MS SQL connections but I could be wrong. The strange thing is, restarting the MS SQL server might be (we're not certain) a solution to get Zope responding again. My theory was that the threads get too attached to a network service that doesn't time out or raise an error. Sadly I can't reproduce it but the next time it happens I'll look at both DeadlockDebugger and the Control_Panel Debug info if there are any clues there. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
On Jan 22, 2008, at 23:33 , Peter Bengtsson wrote: On 22/01/2008, Dieter Maurer [EMAIL PROTECTED] wrote: Peter Bengtsson wrote at 2008-1-22 15:23 +: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason. By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma. This symptom, especially when these hanging threads cause no load at all on the system, certainly point to connections to outside systems that do not respond for some reason. I have seen this issue with Zope instances that make HTTP calls to other systems, or even with a hanging LDAP server which accepted connections from Zope but never returned. Any software that makes outgoing network connections from Zope is a potential suspect. GDB can help you determine where the hanging condition originates. Here's some links: http://plone.org/documentation/how-to/use-gdb-to-get-backtraces-of-hung-spinning-sites http://zopelabs.com/cookbook/1073504990 jens ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
- Original Message - From: Peter Bengtsson [EMAIL PROTECTED] To: Dieter Maurer [EMAIL PROTECTED] Cc: zope user list zope@zope.org Sent: Tuesday, January 22, 2008 5:33 PM Subject: Re: [Zope] Counting threads On 22/01/2008, Dieter Maurer [EMAIL PROTECTED] wrote: Peter Bengtsson wrote at 2008-1-22 15:23 +: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason. By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma. Sadly I can't reproduce it but the next time it happens I'll look at both DeadlockDebugger and the Control_Panel Debug info if there are any clues there. You could also try this: http://www.zopelabs.com/cookbook/1073504990 Jonathan ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Re: [Zope] Counting threads
- Original Message - From: Peter Bengtsson [EMAIL PROTECTED] On 22/01/2008, Dieter Maurer [EMAIL PROTECTED] wrote: Peter Bengtsson wrote at 2008-1-22 15:23 +: Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. I'm just not certain if it's this. It seems highly related to MS SQL connections but I could be wrong. The strange thing is, restarting the MS SQL server might be (we're not certain) a solution to get Zope responding again. My theory was that the threads get too attached to a network service that doesn't time out or raise an error. You have absolutelly true! I was the same problems in past. I was make up to 150 threads in some rare cases, but 50..100 in general if I use MSSQL. It was with Zope 2.5.1, with modern zope, see attachment about pool_size. Now I use 80 threads and 60 pool_size and my Zope+MSSQL is fine. MSSQL hits in day peaks 100% CPU load (about hour, two) , but it is not my business. But my business is functional zope webs which eats a lot of RAM due caching policy - about 5..10 times more than before... So you have been pointed and warned ;-) Best Regards, JL. pool_size.txt Description: Binary data ___ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
