RE: Hung threads
Konstantin - I assume that the APR code is responsible for issuing that call? The reason I'm asking is that I moved the customer to a new server over the weekend and they showed no sign of the problem on the new setup (Windows 2008R2*, Sun JDK 1.6.0_20 x64, Tomcat 5.5.27, native lib 1.1.16). That setup was chosen because it closely matches another server that was not displaying the problem (diffs: non-R2 2008, JDK 1.6.0_13). Over last night, a normal Monday in SE ASIA, they only generated 14 worker threads. I'll continue monitoring throughout the week to see if this holds. Since I still have the box, and have another similar setup I can use for testing, I'll see if I can duplicate the issues, at least the creeping thread leak, on those. But for now, I've got my production setup back stable. I'm not completely ruling out a box/OS issue on the old hardware, but it had been running fine for at least a year. The box had started having some odd behavior (appear to hang until you hit a key on the keyboard, then blue screen - out of non-paged memory), that I traced down to a recurring WinMgmt .NET performance counter error that had started showing up in the Application Event logs every couple of minutes. I removed those counters from the performance monitoring and the system appeared to stabilize. Later, I had rasctrs error messages appear every few seconds. Applying OS updates fixed that. In both cases, the error message text could not be determined from the dlls that should have had them. (It had also halted 3 times since November with a CPU error - only logged by the Dell server mgmt, not Windows - so we had made the decision to replace the box.) I had updated the JDK native lib after the fixing the WinMgmt problem, and that's when I started see the thread leak. I backed out the JDK upgrade, but still saw the leak. I backed out the JDK, but never got a night's production out of it before new server arrived. Jeff -Original Message- From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Sent: Friday, April 16, 2010 4:10 PM To: Tomcat Users List Subject: Re: Hung threads 2010/4/16 Jeffrey Janner jeffrey.jan...@polydyne.com Well, it's definitely deadlocking once it hits the maxThreads limit. The worker is returned to the workers pool by an AprEndpoint.recycleWorkerThread(..) call. If that call is skipped, the workers will leak. That is theoretically. I do not remember any reports on such an issue. Do you have any error messages in your logs, or in the catalina.out file (aka the file that catches your stderr stream)? Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
Chris - I've noticed that little attachment issue also, but it only applies to replies to your posts. It doesn’t appear that way when I send it. (BTW: I'm using Outlook 2007 via Exchange 2003.) I didn't think that our code would be the cause, but since I've not done any coding for Tomcat, I wasn't sure. I'm thinking it might be an issue with the 1.1.18 native library. None of my other systems are showing the issue, but they are using version 1.1.3 of the library, although one is using 1.1.16 and not showing the problem either. I backed them back down to the 1.1.3 version and am waiting to see if the problem re-presents itself. Finally, could it be something the end user's browser is doing/not doing? Or perhaps an intervening proxy at the customer site? I haven't been able to reproduce the issue using in-house browsers/users against the production site, or even a similarly configured test site. Which side of the conversation actually closes the connection? And leaving the connection open result in threads being busy, even though they are waiting? Jeff -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, April 16, 2010 3:50 PM To: Tomcat Users List Subject: Re: Hung threads Jeffrey, BTW your mailer and/or mail exchanger sucks... it puts your message into the html-only portion of a multipart/alternative message which means that basically your messages don't show up properly. Instead, I see the confidential warning that gets attached to all your messages, sometimes (if I'm lucky) followed by your actual message. Usually, I have to open the attachment that contains your actual text. :( On 4/15/2010 12:57 PM, Jeffrey Janner wrote: I've not got the tools nor training to do a Windows build. The build is not Windows-specific: it's just compiling the Java classes, along with a source patch that you could probably hand-enter given the diff if you don't have a win32 'patch' binary handy. I could gather the tools, but I'd rather not learn by patching a production system. Sounds like a good idea: can you set up a test system that you can pound while you figure out the magic recipe of versions that will yield a stable system? I think it's probably a sudden usage spike by the customer's 3rd-party users. I can increase their max thread count and probably give them relief. Plus then I can really see if there is a possible connection leak going on in my developer's code. It's unlikely that your code is leaking HTTP connections somehow. Maybe JDBC connections, but that's another story. - -chris *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments.
RE: Hung threads
NO errors logged anywhere. That's the really frustrating part of it. That and that I cannot reproduce it using in-house clients against the server during the day. I haven't bothered to crank up to debug either, since it's a production system. I want to lean to there being an issue with 1.1.18 native, since that's the only recent change that was still in effect. I backed that down to the previous version, but the end users haven't used the system yet to check if that is the issue. I do have a 1.1.16 version on a system that does not show the problem. -Original Message- From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Sent: Friday, April 16, 2010 4:10 PM To: Tomcat Users List Subject: Re: Hung threads 2010/4/16 Jeffrey Janner jeffrey.jan...@polydyne.com Well, it's definitely deadlocking once it hits the maxThreads limit. The worker is returned to the workers pool by an AprEndpoint.recycleWorkerThread(..) call. If that call is skipped, the workers will leak. That is theoretically. I do not remember any reports on such an issue. Do you have any error messages in your logs, or in the catalina.out file (aka the file that catches your stderr stream)? Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
And the solution to the attachment issue is removing the PGP signing stuff from your message in my reply. -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Saturday, April 17, 2010 1:04 PM To: Tomcat Users List Subject: RE: Hung threads Chris - I've noticed that little attachment issue also, but it only applies to replies to your posts. It doesn’t appear that way when I send it. (BTW: I'm using Outlook 2007 via Exchange 2003.) I didn't think that our code would be the cause, but since I've not done any coding for Tomcat, I wasn't sure. I'm thinking it might be an issue with the 1.1.18 native library. None of my other systems are showing the issue, but they are using version 1.1.3 of the library, although one is using 1.1.16 and not showing the problem either. I backed them back down to the 1.1.3 version and am waiting to see if the problem re-presents itself. Finally, could it be something the end user's browser is doing/not doing? Or perhaps an intervening proxy at the customer site? I haven't been able to reproduce the issue using in-house browsers/users against the production site, or even a similarly configured test site. Which side of the conversation actually closes the connection? And leaving the connection open result in threads being busy, even though they are waiting? Jeff -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Friday, April 16, 2010 3:50 PM To: Tomcat Users List Subject: Re: Hung threads Jeffrey, BTW your mailer and/or mail exchanger sucks... it puts your message into the html-only portion of a multipart/alternative message which means that basically your messages don't show up properly. Instead, I see the confidential warning that gets attached to all your messages, sometimes (if I'm lucky) followed by your actual message. Usually, I have to open the attachment that contains your actual text. :( On 4/15/2010 12:57 PM, Jeffrey Janner wrote: I've not got the tools nor training to do a Windows build. The build is not Windows-specific: it's just compiling the Java classes, along with a source patch that you could probably hand-enter given the diff if you don't have a win32 'patch' binary handy. I could gather the tools, but I'd rather not learn by patching a production system. Sounds like a good idea: can you set up a test system that you can pound while you figure out the magic recipe of versions that will yield a stable system? I think it's probably a sudden usage spike by the customer's 3rd-party users. I can increase their max thread count and probably give them relief. Plus then I can really see if there is a possible connection leak going on in my developer's code. It's unlikely that your code is leaking HTTP connections somehow. Maybe JDBC connections, but that's another story. - -chris *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments.
RE: Hung threads
*** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. ---BeginMessage--- Well, it's definitely deadlocking once it hits the maxThreads limit. I had this instance and another one on the same machine both hit the max last night and the acceptor stopped waiting on workers. All the workers show their normal wait state, and some of the Pollers showed they were waiting on workers also. What I appear to be seeing is that the requests come in and get processed on a minimal number of workers, but at some point, new workers are getting created until we reach the maxThreads limit (150). I'm not seeing a traffic load that indicates that it should ever be using that many workers, there are only about 15-20 end users in the app at any one time. I'm not seeing this on other systems running the same version of the app tomcat, although they are running a much older native library. I've been monitoring the instance using jconsole. On the instance that died early last night, they were sitting at about 135 worker threads at 8:00am and stayed there all day, even with some cursory logging in and checking a few pages during the day. Then, at about 7:00pm, when the Asia user based started working, the Threads graph in jconsole starts climbing, levels for about 30 minutes, and then climbed the rest of the way until it hit the 150 limit. Almost as though it wasn't using the bulk of the existing worker threads. The resulting threads looked just like those in the previous thread dump -- all workers in normal wait state, but the Acceptor waiting on workers. Question: Could there be something that our programmers are doing (or more likely NOT doing) that would allow workers to return to waiting, but not actually be free for work? Otherwise, I'd have to assume something in the native library is mucking up the thread pool count. Jeff -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, April 15, 2010 11:42 AM To: Tomcat Users List Subject: Re: Hung threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Konstantin, On 4/15/2010 10:19 AM, Konstantin Kolinko wrote: +1. If it is stuck there, it will not accept any more incoming requests. Thanks for the confirmation that Jeffrey is deadlocked. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 Heh. This guy is bouncing from one bug to the next, here. Sorry, Jeffrey. :( A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Jeffrey, do you have the inclination to apply this patch to your TC instance? Compiling TC 5.5 was relatively simply IIRC. Or, maybe someone would be willing to roll you a binary patch. - -chris ---End Message--- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey, BTW your mailer and/or mail exchanger sucks... it puts your message into the html-only portion of a multipart/alternative message which means that basically your messages don't show up properly. Instead, I see the confidential warning that gets attached to all your messages, sometimes (if I'm lucky) followed by your actual message. Usually, I have to open the attachment that contains your actual text. :( On 4/15/2010 12:57 PM, Jeffrey Janner wrote: I've not got the tools nor training to do a Windows build. The build is not Windows-specific: it's just compiling the Java classes, along with a source patch that you could probably hand-enter given the diff if you don't have a win32 'patch' binary handy. I could gather the tools, but I'd rather not learn by patching a production system. Sounds like a good idea: can you set up a test system that you can pound while you figure out the magic recipe of versions that will yield a stable system? I think it's probably a sudden usage spike by the customer's 3rd-party users. I can increase their max thread count and probably give them relief. Plus then I can really see if there is a possible connection leak going on in my developer's code. It's unlikely that your code is leaking HTTP connections somehow. Maybe JDBC connections, but that's another story. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvIzYUACgkQ9CaO5/Lv0PDY4gCfZQunGew3JAWbESHOZvzPsmJi QCEAn0oUmQLv8ulOATqg68vFPB/0NU2j =/77Z -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
2010/4/16 Jeffrey Janner jeffrey.jan...@polydyne.com Well, it's definitely deadlocking once it hits the maxThreads limit. The worker is returned to the workers pool by an AprEndpoint.recycleWorkerThread(..) call. If that call is skipped, the workers will leak. That is theoretically. I do not remember any reports on such an issue. Do you have any error messages in your logs, or in the catalina.out file (aka the file that catches your stderr stream)? Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
2010/4/13 Christopher Schultz ch...@christopherschultz.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff, Okay, you're running into the limits of my knowledge of the APR connector's internals. On 4/13/2010 1:24 PM, Jeffrey Janner wrote: http-172.16.27.1-443-Acceptor-0 daemon prio=6 tid=0x6425c400 nid=0xb6c4 in Object.wait() [0x65d7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint.getWorkerThread(AprEndpoint.java:924) - locked 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:997) at java.lang.Thread.run(Thread.java:619) This is the acceptor thread which basically accepts all the incoming requests and assigns them to worker threads: it's just a traffic cop. This thread looks like it's stuck waiting for an available worker thread (AprEndpoint.getWorkerThread) so it looks like something is definitely wrong, here. +1. If it is stuck there, it will not accept any more incoming requests. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Unfortunately for you, you'll need someone like Filip who knows everything about these Connectors to comment. Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 It looks like a thread called http-172.16.27.1-443-150 exists, which suggests that all maxThreads have been allocated, though they really do all look idle to me. Hmm. Yeah, wait for Filip. :) Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Konstantin, On 4/15/2010 10:19 AM, Konstantin Kolinko wrote: +1. If it is stuck there, it will not accept any more incoming requests. Thanks for the confirmation that Jeffrey is deadlocked. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 Heh. This guy is bouncing from one bug to the next, here. Sorry, Jeffrey. :( A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Jeffrey, do you have the inclination to apply this patch to your TC instance? Compiling TC 5.5 was relatively simply IIRC. Or, maybe someone would be willing to roll you a binary patch. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvHQcwACgkQ9CaO5/Lv0PA+wgCghM9Vcpn550UWN4EaQ38vTbaT eTEAniwPB4PsiNNpHMJd/ZMiN7Ai/Nko =+lIJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
Thanks Konstantin - That's a possibility. Is it feasible that all 150 worker threads would enter await before the Acceptor thread got some CPU time again? I guess so. I suppose setting the priority of the Acceptor thread higher than the workers would minimize the chances of this happening. I dropped them back to a 1.5 JVM and made it through the night last night, though that might have just been lower load on the system as well. It only opened about 135 worker threads. Hopefully, 5.5.30 will be along faster than 5.5.29. I'm still waiting on whomever to put the Windows installer version out there for that. Jeff -Original Message- From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Sent: Thursday, April 15, 2010 9:20 AM To: Tomcat Users List Subject: Re: Hung threads 2010/4/13 Christopher Schultz ch...@christopherschultz.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff, Okay, you're running into the limits of my knowledge of the APR connector's internals. On 4/13/2010 1:24 PM, Jeffrey Janner wrote: http-172.16.27.1-443-Acceptor-0 daemon prio=6 tid=0x6425c400 nid=0xb6c4 in Object.wait() [0x65d7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint.getWorkerThread(AprEndpoint.java:924) - locked 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:997) at java.lang.Thread.run(Thread.java:619) This is the acceptor thread which basically accepts all the incoming requests and assigns them to worker threads: it's just a traffic cop. This thread looks like it's stuck waiting for an available worker thread (AprEndpoint.getWorkerThread) so it looks like something is definitely wrong, here. +1. If it is stuck there, it will not accept any more incoming requests. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Unfortunately for you, you'll need someone like Filip who knows everything about these Connectors to comment. Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 It looks like a thread called http-172.16.27.1-443-150 exists, which suggests that all maxThreads have been allocated, though they really do all look idle to me. Hmm. Yeah, wait for Filip. :) Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
*** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. ---BeginMessage--- I've not got the tools nor training to do a Windows build. I could gather the tools, but I'd rather not learn by patching a production system. I think it's probably a sudden usage spike by the customer's 3rd-party users. I can increase their max thread count and probably give them relief. Plus then I can really see if there is a possible connection leak going on in my developer's code. Thanks for the sympathy. Jeff -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Thursday, April 15, 2010 11:42 AM To: Tomcat Users List Subject: Re: Hung threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Konstantin, On 4/15/2010 10:19 AM, Konstantin Kolinko wrote: +1. If it is stuck there, it will not accept any more incoming requests. Thanks for the confirmation that Jeffrey is deadlocked. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 Heh. This guy is bouncing from one bug to the next, here. Sorry, Jeffrey. :( A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Jeffrey, do you have the inclination to apply this patch to your TC instance? Compiling TC 5.5 was relatively simply IIRC. Or, maybe someone would be willing to roll you a binary patch. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvHQcwACgkQ9CaO5/Lv0PA+wgCghM9Vcpn550UWN4EaQ38vTbaT eTEAniwPB4PsiNNpHMJd/ZMiN7Ai/Nko =+lIJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org ---End Message--- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
On 4/15/2010 12:47 PM, Jeffrey Janner wrote: Thanks Konstantin - That's a possibility. Is it feasible that all 150 worker threads would enter await before the Acceptor thread got some CPU time again? I guess so. I suppose setting the priority of the Acceptor thread higher than the workers would minimize the chances of this happening. I dropped them back to a 1.5 JVM and made it through the night last night, though that might have just been lower load on the system as well. It only opened about 135 worker threads. Hopefully, 5.5.30 will be along faster than 5.5.29. I'm still waiting on whomever to put the Windows installer version out there for that. Jeff If you already have an installation, you don't need a windows installation package to install the updates. Just unzip the package on top of your old installation, and everything will be updated (assuming it's the same base version, such as upgrading from 5.5.20 to 5.5.30). -Original Message- From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Sent: Thursday, April 15, 2010 9:20 AM To: Tomcat Users List Subject: Re: Hung threads 2010/4/13 Christopher Schultzch...@christopherschultz.net: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff, Okay, you're running into the limits of my knowledge of the APR connector's internals. On 4/13/2010 1:24 PM, Jeffrey Janner wrote: http-172.16.27.1-443-Acceptor-0 daemon prio=6 tid=0x6425c400 nid=0xb6c4 in Object.wait() [0x65d7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint.getWorkerThread(AprEndpoint.java:924) - locked0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:997) at java.lang.Thread.run(Thread.java:619) This is the acceptor thread which basically accepts all the incoming requests and assigns them to worker threads: it's just a traffic cop. This thread looks like it's stuck waiting for an available worker thread (AprEndpoint.getWorkerThread) so it looks like something is definitely wrong, here. +1. If it is stuck there, it will not accept any more incoming requests. It might be that you bumped into BZ 48843 https://issues.apache.org/bugzilla/show_bug.cgi?id=48843 A patch for it is already available, proposed, and has enough votes, so it will be applied shortly. That will be 5.5.30, though. Unfortunately for you, you'll need someone like Filip who knows everything about these Connectors to comment. Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 It looks like a thread called http-172.16.27.1-443-150 exists, which suggests that all maxThreads have been allocated, though they really do all look idle to me. Hmm. Yeah, wait for Filip. :) Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
Before someone asks for it: Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 connectionTimeout=2 disableUploadTimeout=true compression=on compressableMimeType=text/html,text/xml,text/plain,text/css,text/csv,te xt/javascript,text/rtf,text/richtext scheme=https secure=true clientAuth=false SSLEngine=on SSLCertificateFile=path_to_server.crt SSLCertificateKeyFile= path_to_server.key SSLCertificateChainFile=path_to_server_chain.crt SSLPassword=changeit / -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Tuesday, April 13, 2010 9:44 AM To: Tomcat Users List Subject: Hung threads I had a connector go from minimal connections (restart) to all threads hung overnight. The current thread dump shows the threads as follows: http-172.16.27.1-443-150 daemon prio=6 tid=0x66793800 nid=0xe43c in Object.wait() [0x7063f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked 0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-149 daemon prio=6 tid=0x6496d400 nid=0xe400 in Object.wait() [0x705ff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked 0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-148 daemon prio=6 tid=0x6496f400 nid=0xe3b0 in Object.wait() [0x705bf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked 0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) and so on. Other connectors are running OK, though some also show threads in the same state. All other threads look normal. This has one customer's app hung, but others in the Tomcat instance are working just fine. Any help on where to start looking here? Server Details: Windows 2000 SP4, Sun JDK 1.6.0_18, Tomcat 5.5.17, native libs 1.1.18. Waiting for TC 5.5.29 before upgrading due to bugs with APR/SSL in 5.5.28. Jeff *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey, On 4/13/2010 10:44 AM, Jeffrey Janner wrote: I had a connector go from minimal connections (restart) to all threads hung overnight. The current thread dump shows the threads as follows: http-172.16.27.1-443-150 daemon prio=6 tid=0x66793800 nid=0xe43c in Object.wait() [0x7063f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) I'm pretty sure these threads are considered idle. Are you able to make a request on port 443 and have it serviced? Other connectors are running OK, though some also show threads in the same state. All other threads look normal. This has one customer's app hung, but others in the Tomcat instance are working just fine. Any help on where to start looking here? The connectors are global for the entire app server, so if any webapps are accessible, they should all be accessible (unless this is the only webapp that requires SSL, in which case a dead SSL connector would certainly affect only that one webapp). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvEkWUACgkQ9CaO5/Lv0PAuIgCfULZw2F0zb+CBMWvJVUC8JIHt iuMAoJM6Cs2dC7ufRfEA/13Z513NjfdZ =WSuM -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
Looks like just idle threads, not really a problem If you want the thread pool to shrink and stop idle threads, use an Executor element in server.xml Filip On 04/13/2010 08:50 AM, Jeffrey Janner wrote: Before someone asks for it: Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 connectionTimeout=2 disableUploadTimeout=true compression=on compressableMimeType=text/html,text/xml,text/plain,text/css,text/csv,te xt/javascript,text/rtf,text/richtext scheme=https secure=true clientAuth=false SSLEngine=on SSLCertificateFile=path_to_server.crt SSLCertificateKeyFile= path_to_server.key SSLCertificateChainFile=path_to_server_chain.crt SSLPassword=changeit / -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Tuesday, April 13, 2010 9:44 AM To: Tomcat Users List Subject: Hung threads I had a connector go from minimal connections (restart) to all threads hung overnight. The current thread dump shows the threads as follows: http-172.16.27.1-443-150 daemon prio=6 tid=0x66793800 nid=0xe43c in Object.wait() [0x7063f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-149 daemon prio=6 tid=0x6496d400 nid=0xe400 in Object.wait() [0x705ff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-148 daemon prio=6 tid=0x6496f400 nid=0xe3b0 in Object.wait() [0x705bf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) and so on. Other connectors are running OK, though some also show threads in the same state. All other threads look normal. This has one customer's app hung, but others in the Tomcat instance are working just fine. Any help on where to start looking here? Server Details: Windows 2000 SP4, Sun JDK 1.6.0_18, Tomcat 5.5.17, native libs 1.1.18. Waiting for TC 5.5.29 before upgrading due to bugs with APR/SSL in 5.5.28. Jeff *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination,
RE: Hung threads
*** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. ---BeginMessage--- Hi Chris - Sorry I was extremely clear. My config has each customer in a separate service structure, so each get his own set of connectors (IPs). Yes, I am unable to get a response on this specific IP/port combo. All other connectors were responding and all of those apps were working fine, from what I could tell. The Tomcat Manager only showed 4 sessions for this host when I started, and that eventually dropped to 1, but I could never get a response from the app. After a restart, I did a connect, and immediately had 7 threads start up and apparently end in the same state (at least it looks the same in jconsole). I was able to navigate the app and never had the thread count increase. Every check on the thread states showed the same, though the Total blocked/Total Waited counters keep increasing. The really interesting thing is that over more than an hour with virtually no traffic, should not some of the threads started shutting down? Jeff -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, April 13, 2010 10:45 AM To: Tomcat Users List Subject: Re: Hung threads -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey, On 4/13/2010 10:44 AM, Jeffrey Janner wrote: I had a connector go from minimal connections (restart) to all threads hung overnight. The current thread dump shows the threads as follows: http-172.16.27.1-443-150 daemon prio=6 tid=0x66793800 nid=0xe43c in Object.wait() [0x7063f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) I'm pretty sure these threads are considered idle. Are you able to make a request on port 443 and have it serviced? Other connectors are running OK, though some also show threads in the same state. All other threads look normal. This has one customer's app hung, but others in the Tomcat instance are working just fine. Any help on where to start looking here? The connectors are global for the entire app server, so if any webapps are accessible, they should all be accessible (unless this is the only webapp that requires SSL, in which case a dead SSL connector would certainly affect only that one webapp). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvEkWUACgkQ9CaO5/Lv0PAuIgCfULZw2F0zb+CBMWvJVUC8JIHt iuMAoJM6Cs2dC7ufRfEA/13Z513NjfdZ =WSuM -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org ---End Message--- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff, On 4/13/2010 12:10 PM, Jeffrey Janner wrote: Sorry I was extremely clear. My config has each customer in a separate service structure, so each get his own set of connectors (IPs). Gotcha. Yes, I am unable to get a response on this specific IP/port combo. That's not good. Do you get a connection refused, or do you get a timeout? The Tomcat Manager only showed 4 sessions for this host when I started, and that eventually dropped to 1, but I could never get a response from the app. Note that lots of requests can be services without a session being created (depending on your webapp). After a restart, I did a connect, and immediately had 7 threads start up and apparently end in the same state (at least it looks the same in jconsole). The state of the threads in your thread dump is normal: WAITING means blocked on a monitor (i.e. Object.wait). Those threads are waiting to be notified by the acceptor thread that a request has been made and is ready for processing. I was able to navigate the app and never had the thread count increase. Every check on the thread states showed the same, though the Total blocked/Total Waited counters keep increasing. That's strange. The really interesting thing is that over more than an hour with virtually no traffic, should not some of the threads started shutting down? Read Filip's post about using an Executor to remove unnsed threads: the standard Connector-based thread pooling will never remove threads, even though it looks like they will based upon the configuration of the Connector. Executors are much better for resource management. I'm still concerned that you are unable to get a request through to the connector. Can you double-check that you haven't grabbed the wrong section of the thread dump? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvEmroACgkQ9CaO5/Lv0PAx9wCfY2cyUtTCRxDGHWH6H45KZ05L +80An25jy/aVgJ6LSdprOJI+Kcd0hbLK =N6cI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
Thanks, but Executor is not available in 5.5 -- at least according to the documentation. -Original Message- From: Filip Hanik - Dev Lists [mailto:devli...@hanik.com] Sent: Tuesday, April 13, 2010 11:03 AM To: Tomcat Users List Subject: Re: Hung threads Looks like just idle threads, not really a problem If you want the thread pool to shrink and stop idle threads, use an Executor element in server.xml Filip On 04/13/2010 08:50 AM, Jeffrey Janner wrote: Before someone asks for it: Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 connectionTimeout=2 disableUploadTimeout=true compression=on compressableMimeType=text/html,text/xml,text/plain,text/css,text/csv,te xt/javascript,text/rtf,text/richtext scheme=https secure=true clientAuth=false SSLEngine=on SSLCertificateFile=path_to_server.crt SSLCertificateKeyFile= path_to_server.key SSLCertificateChainFile=path_to_server_chain.crt SSLPassword=changeit / -Original Message- From: Jeffrey Janner [mailto:jeffrey.jan...@polydyne.com] Sent: Tuesday, April 13, 2010 9:44 AM To: Tomcat Users List Subject: Hung threads I had a connector go from minimal connections (restart) to all threads hung overnight. The current thread dump shows the threads as follows: http-172.16.27.1-443-150 daemon prio=6 tid=0x66793800 nid=0xe43c in Object.wait() [0x7063f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a924f00 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-149 daemon prio=6 tid=0x6496d400 nid=0xe400 in Object.wait() [0x705ff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a954140 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-148 daemon prio=6 tid=0x6496f400 nid=0xe3b0 in Object.wait() [0x705bf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Worker.await(AprEndpoint.java:125 5) - locked0x1a8655f0 (a org.apache.tomcat.util.net.AprEndpoint$Worker) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1280) at java.lang.Thread.run(Thread.java:619) and so on. Other connectors are running OK, though some also show threads in the same state. All other threads look normal. This has one customer's app hung, but others in the Tomcat instance are working just fine. Any help on where to start looking here? Server Details: Windows 2000 SP4, Sun JDK 1.6.0_18, Tomcat 5.5.17, native libs 1.1.18. Waiting for TC 5.5.29 before upgrading due to bugs with APR/SSL in 5.5.28. Jeff *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. *** NOTICE * This message is intended for the use of the individual
RE: Hung threads
I got no response. No error from Tomcat or the browser. In IE, the status bar progress bar runs a bit, then goes to the end and disappears. The page doesn't change. I had to use jstack to get the dump. I just posted a sample of a few of the connectors. I've included the entire dump file as an attachment to this post (if it doesn't get stripped). But I didn't see anything sitting in our code. Note that the 172.16.27.1 entries are for the hung host. The other IP address is for the working host. Both hosts are configured to send all traffic to HTTPS inside the app. Unfortunately, Executor doesn't appear to be available in 5.5.x. And before the inevitable, the available heap looked just fine. -Original Message- From: Christopher Schultz [mailto:ch...@christopherschultz.net] Sent: Tuesday, April 13, 2010 11:24 AM To: Tomcat Users List Subject: Re: Hung threads Jeff, On 4/13/2010 12:10 PM, Jeffrey Janner wrote: Sorry I was extremely clear. My config has each customer in a separate service structure, so each get his own set of connectors (IPs). Gotcha. Yes, I am unable to get a response on this specific IP/port combo. That's not good. Do you get a connection refused, or do you get a timeout? The Tomcat Manager only showed 4 sessions for this host when I started, and that eventually dropped to 1, but I could never get a response from the app. Note that lots of requests can be services without a session being created (depending on your webapp). After a restart, I did a connect, and immediately had 7 threads start up and apparently end in the same state (at least it looks the same in jconsole). The state of the threads in your thread dump is normal: WAITING means blocked on a monitor (i.e. Object.wait). Those threads are waiting to be notified by the acceptor thread that a request has been made and is ready for processing. I was able to navigate the app and never had the thread count increase. Every check on the thread states showed the same, though the Total blocked/Total Waited counters keep increasing. That's strange. The really interesting thing is that over more than an hour with virtually no traffic, should not some of the threads started shutting down? Read Filip's post about using an Executor to remove unnsed threads: the standard Connector-based thread pooling will never remove threads, even though it looks like they will based upon the configuration of the Connector. Executors are much better for resource management. I'm still concerned that you are unable to get a request through to the connector. Can you double-check that you haven't grabbed the wrong section of the thread dump? - -chris *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeffrey, On 4/13/2010 12:46 PM, Jeffrey Janner wrote: http-172.16.27.1-443-Poller-7 daemon prio=6 tid=0x649cnVVpEWVZadmQxSnJTbEpXDQpWVnBFV1ZaYWRtUXhTbkpUYkVwWERRcFdWbkJGVjFaYVlXUnRVWGhU YmtwVVlrVndXRVJSY0ZkV2JrSkdWakZhWVZsWFVuUlZXR2hVDQpZbXR3VlZsclZuZFhSVkpTWTBa a1YySnJTa2RXYWtaaFdWWnNXRlZ1VWxaWFIyaFZEUXBaYlhSM1ZsWnNjbFp1WkZoU1ZrcFRXVEJh DQphMVl5U25KVGEyUlhZV3RhYUZkV1duTlhSbFoxVld4YVdGSXlhRlpFVVhCYVlsaFNNMVpzV25O amJGcDFXa1pvVTFacmNGUlhWRUpoDQpEUXBoTVZsNVUyNUtWR0V5VWxoWlYzUmhZVVprVjFkdVRs aFNiRm94VmxkNFlWZEdTWGxoUmxwRlZWaENZVmxzYUZOTk1WcHpWMjVPDQphbUpHY0RGWGExcHZW VEZhY21OR1VsaFdSVXBvRFFwRVVYQm9UVlpzTlZVeU5VdFdSMFY1Vld4b1dsWXpVbWhaVlZwclZq RmtkVlJzDQphRk5pUm05NFZteGtORmxXWkVkVFdHeG9VbXh3UmxaV2FFTlpWbXh6WVVaT1RrMVdj SHBXTWpWUERRcGhiVXBIWTBSR1dHRXhjSFpXDQpWRVpoWTIxT1IxVnNhRmRTVlhCdlJGRndSVlZZ UW05VVZscHpUbFpWZVU1VmRGZFNNRlkxVmxkNGIxZHNXWHBWYldoYVZsWndjbFpxDQpSbXRrVmxK ekRRcGhSazVwVW0wNU5GWnRlR3RPUm14WFdrVmtWRmRIZUc5VmJYaDNVbXhhVjJGRlRscFdiWGg2 Now that certainly looks weird. Can you check again and re-post the poller thread's status? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvEotQACgkQ9CaO5/Lv0PDEFgCZAb/KLuHKT1Z2EDldrOQ59tCg RoEAoJF53thc11WHpXI7ipDmoXumT5Si =PsHo -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Hung threads
You're right, that was weird. Not sure what did that. It wasn't that way in the original file. And it wasn't that way before I hit send. Here's the rest of the dump, picking up from just before encoding: RMI Scheduler(0) daemon prio=6 tid=0x64885400 nid=0xaa8c waiting on condition [0x6707f000] java.lang.Thread.State: TIMED_WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for 0x0b738a30 (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1963) at java.util.concurrent.DelayQueue.take(DelayQueue.java:164) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:583) at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:576) at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-7 daemon prio=6 tid=0x649c5800 nid=0xb7a8 in Object.wait() [0x65fbf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96a938 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1127) - locked 0x0b96a938 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-6 daemon prio=6 tid=0x649f5c00 nid=0xaef4 in Object.wait() [0x65f7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint.getWorkerThread(AprEndpoint.java:924) - locked 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1165) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-5 daemon prio=6 tid=0x64758400 nid=0xb5dc in Object.wait() [0x65f3f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96a998 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1127) - locked 0x0b96a998 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-4 daemon prio=6 tid=0x637f0400 nid=0xb6fc in Object.wait() [0x65eff000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96a9c8 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1127) - locked 0x0b96a9c8 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-3 daemon prio=6 tid=0x64902400 nid=0xb27c in Object.wait() [0x65ebf000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96a9f8 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1127) - locked 0x0b96a9f8 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-2 daemon prio=6 tid=0x63848400 nid=0xb364 in Object.wait() [0x65e7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96aa28 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint$Poller.run(AprEndpoint.java:1127) - locked 0x0b96aa28 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at java.lang.Thread.run(Thread.java:619) http-172.16.27.1-443-Poller-1 daemon prio=6 tid=0x64505400 nid=0xb674 in Object.wait() [0x65e3f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b96aa58 (a org.apache.tomcat.util.net.AprEndpoint$Poller) at
Re: Hung threads
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jeff, Okay, you're running into the limits of my knowledge of the APR connector's internals. On 4/13/2010 1:24 PM, Jeffrey Janner wrote: http-172.16.27.1-443-Acceptor-0 daemon prio=6 tid=0x6425c400 nid=0xb6c4 in Object.wait() [0x65d7f000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.AprEndpoint.getWorkerThread(AprEndpoint.java:924) - locked 0x0b8bafd8 (a org.apache.tomcat.util.net.AprEndpoint$WorkerStack) at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:997) at java.lang.Thread.run(Thread.java:619) This is the acceptor thread which basically accepts all the incoming requests and assigns them to worker threads: it's just a traffic cop. This thread looks like it's stuck waiting for an available worker thread (AprEndpoint.getWorkerThread) so it looks like something is definitely wrong, here. Unfortunately for you, you'll need someone like Filip who knows everything about these Connectors to comment. Connector address=172.16.27.1 port=443 maxHttpHeaderSize=8192 maxThreads=150 minSpareThreads=5 maxSpareThreads=75 enableLookups=false acceptCount=100 It looks like a thread called http-172.16.27.1-443-150 exists, which suggests that all maxThreads have been allocated, though they really do all look idle to me. Hmm. Yeah, wait for Filip. :) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvEsBgACgkQ9CaO5/Lv0PCV9wCgtniW8WPI9nFm86jHxt1zPc5i kOMAnR6loJp236+HzeFER9ULkYM6TW8W =jxfF -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
