Re: Tomcat current thread count
Hi Charles, Is there any way I can clear the current thread count without restarting tomcat? The programmer is unable to accept the problem so it seems I will have to find a way to clear current thread number. Thank you. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Mon, Jun 28, 2010 at 3:26 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Here we go again. Same comments apply as before: You have numerous Timer-x threads going, which makes me suspicious that your webapp is starting up extra threads and not properly managing them. Simply making them daemon (as they are) is not sufficient, since the daemon threads won't terminate until Tomcat is fully shut down. There are also some other threads running that start in your webapp, including some waiting for input on UDP sockets. This still looks like poor resource management by your webapp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat current thread count
From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Is there any way I can clear the current thread count without restarting tomcat? Clearing the thread *count* does nothing - you have to actually terminate the excess threads. The programmer is unable to accept the problem Then you need another programmer; refusing to fix a blatant error isn't exactly a sterling recommendation. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Was just a double check. Thanks for all the help Chuck. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Mon, Jul 5, 2010 at 3:59 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Is there any way I can clear the current thread count without restarting tomcat? Clearing the thread *count* does nothing - you have to actually terminate the excess threads. The programmer is unable to accept the problem Then you need another programmer; refusing to fix a blatant error isn't exactly a sterling recommendation. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Hi, Here we go again. Let`s see if it accepts. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 8:04 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Özgür, On 6/25/2010 12:21 PM, Ozgur Ozdemircili wrote: I probably wont be able to get the one close to crash but the jstack.out in the attached file contains the output while the applicaction is receiving requests. Hope it will be more helpful. The list stripped your attachment. Try giving it a .txt extension. Sounds stupid, but it appears that works sometimes. I'm guessing that not much help will come from a thread dump during some normal processing... you'll just see your code running normally. What usually trips-up threads is deadlock or long-running requests piling up. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwk758ACgkQ9CaO5/Lv0PBnUACfW2XW2+jZc93diac3BrOGzVWh sQQAnjILLLv0n1lX4lFtvN8FPg0cbABy =jm31 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Deadlock Detection: No deadlocks found. Thread 23415: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23242: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23241: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23240: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23239: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23238: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.util.TimerThread.mainLoop() @bci=201, line=509 (Interpreted frame) - java.util.TimerThread.run() @bci=1, line=462 (Interpreted frame) Thread 23237: (state = IN_NATIVE) - java.net.PlainDatagramSocketImpl.receive0(java.net.DatagramPacket) @bci=0 (Interpreted frame) - java.net.PlainDatagramSocketImpl.receive(java.net.DatagramPacket) @bci=2, line=136 (Interpreted frame) - java.net.DatagramSocket.receive(java.net.DatagramPacket) @bci=243, line=712 (Interpreted frame) - com.serhs.wsserhs.udp.ServerUDP.start() @bci=36, line=60 (Interpreted frame) - com.serhs.wsserhs.udp.ServerUDP.run() @bci=1, line=117 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 22895: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run() @bci=10, line=565 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 22894: (state = IN_NATIVE) - java.net.PlainSocketImpl.socketAccept(java.net.SocketImpl) @bci=0 (Interpreted frame) - java.net.PlainSocketImpl.accept(java.net.SocketImpl) @bci=7, line=390 (Interpreted frame) - java.net.ServerSocket.implAccept(java.net.Socket) @bci=60, line=453 (Interpreted frame) - java.net.ServerSocket.accept() @bci=48, line=421 (Interpreted frame) - org.apache.jk.common.ChannelSocket.accept(org.apache.jk.core.MsgContext) @bci=46, line=307 (Interpreted frame) - org.apache.jk.common.ChannelSocket.acceptConnections() @bci=72, line=661 (Interpreted frame) - org.apache.jk.common.ChannelSocket$SocketAcceptor.runIt(java.lang.Object[]) @bci=4, line=872 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=167, line=690 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 22893: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) - org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run() @bci=26, line=662 (Interpreted frame) - java.lang.Thread.run() @bci=11, line=619 (Interpreted frame) Thread 22892: (state = BLOCKED) - java.lang.Object.wait(long) @bci=0 (Interpreted frame) - java.lang.Object.wait() @bci=2, line=485 (Interpreted frame) -
RE: Tomcat current thread count
From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Here we go again. Same comments apply as before: You have numerous Timer-x threads going, which makes me suspicious that your webapp is starting up extra threads and not properly managing them. Simply making them daemon (as they are) is not sufficient, since the daemon threads won't terminate until Tomcat is fully shut down. There are also some other threads running that start in your webapp, including some waiting for input on UDP sockets. This still looks like poor resource management by your webapp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat current thread count
Hi, In our production enviroment I have a weird problem and I am trying to understand if / how can it be fixed. The error is like above: * * *INFO: Maximum number of threads (250) created for connector with address null and port 8080* * * * * * * *# A fatal error has been detected by the Java Runtime Environment:* *#* *# java.lang.OutOfMemoryError: requested 4096000 bytes for GrET in /BUILD_AREA/jdk6_16/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?* *#* *# Internal Error (allocation.inline.hpp:39), pid=15074, tid=326871952* *# Error: GrET in /BUILD_AREA/jdk6_16/hotspot/src/share/vm/utilities/growableArray.cpp* *#* *# JRE version: 6.0_16-b01* *# Java VM: Java HotSpot(TM) Server VM (14.2-b01 mixed mode linux-x86 )* *# An error report file with more information is saved as:* *# /home/oracle/hs_err_pid15074.log* *#* *# If you would like to submit a bug report, please visit:* *# **http://java.sun.com/webapps/bugreport/crash.jsp*http://java.sun.com/webapps/bugreport/crash.jsp *#* * * * * *#* *# A fatal error has been detected by the Java Runtime Environment:* *#* *# java.lang.OutOfMemoryError: requested 4096000 bytes for GrET in /BUILD_AREA/jdk6_16/hotspot/src/share/vm/utilities/growableArray.cpp. Out of swap space?* *#* *# Internal Error (allocation.inline.hpp:39), pid=15074, tid=326871952* *# Error: GrET in /BUILD_AREA/jdk6_16/hotspot/src/share/vm/utilities/growableArray.cpp* *#* *# JRE version: 6.0_16-b01* *# Java VM: Java HotSpot(TM) Server VM (14.2-b01 mixed mode linux-x86 )* *# An error report file with more information is saved as:* *# /home/oracle/hs_err_pid15074.log* *#* *# If you would like to submit a bug report, please visit:* *# **http://java.sun.com/webapps/bugreport/cr*ash.jsphttp://java.sun.com/webapps/bugreport/crash.jsp # The problem or the weirdness here is everytime we restart a tomcat server this number (250) drops down until it reaches 250 again. For example when I compare current thread count with busy threads I see 240 current and 24 busy. How can I have tomcat drop the current thread count? What can be done to stop the error message above. Thank you. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it
Re: Tomcat current thread count
I am on something right now like this. While running, Do a: netstat -n grep [your port number] | grep CALL_WAIT See if you are hung in CALL+WAIT status. --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 4:01 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: 24
Re: Tomcat current thread count
Hi, The command: netstat -n grep 8080 | grep CALL_WAIT gives me nothing, yet when I update CALL_WAIT with TIME_WAIT I can see them. netstat -n grep 8080 | grep TIME_WAIT P.S: Just restarted the servers. As expected the thread counts dropped to normal. Now I am on 43 Current and 20 Busy. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 11:05 AM, Mick Knutson mknut...@baselogic.comwrote: I am on something right now like this. While running, Do a: netstat -n grep [your port number] | grep CALL_WAIT See if you are hung in CALL+WAIT status. --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 4:01 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: 24
Re: Tomcat current thread count
TIME_WAIT is fine. Not an issue. That just means they are ready to take requests. CALL_WAIT is not good, if they stick around. If this is an OOM error, then have you started this server and attached javaVisualVM onto it to see what the threads and memory are doing? --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 5:16 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: Hi, The command: netstat -n grep 8080 | grep CALL_WAIT gives me nothing, yet when I update CALL_WAIT with TIME_WAIT I can see them. netstat -n grep 8080 | grep TIME_WAIT P.S: Just restarted the servers. As expected the thread counts dropped to normal. Now I am on 43 Current and 20 Busy. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 11:05 AM, Mick Knutson mknut...@baselogic.com wrote: I am on something right now like this. While running, Do a: netstat -n grep [your port number] | grep CALL_WAIT See if you are hung in CALL+WAIT status. --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 4:01 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: 24
Re: Tomcat current thread count
On 25.06.2010 11:22, Mick Knutson wrote: TIME_WAIT is fine. Not an issue. That just means they are ready to take requests. CALL_WAIT is not good, if they stick around. If this is an OOM error, then have you started this server and attached javaVisualVM onto it to see what the threads and memory are doing? Not judging on whether that's the right approach, but you probably meant CLOSE_WAIT. There is no TCP state CALL_WAIT. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Im actually using Javamelody to monitor the apps. Yet still nothing. Any ideas ? Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 11:22 AM, Mick Knutson mknut...@baselogic.comwrote: TIME_WAIT is fine. Not an issue. That just means they are ready to take requests. CALL_WAIT is not good, if they stick around. If this is an OOM error, then have you started this server and attached javaVisualVM onto it to see what the threads and memory are doing? --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 5:16 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: Hi, The command: netstat -n grep 8080 | grep CALL_WAIT gives me nothing, yet when I update CALL_WAIT with TIME_WAIT I can see them. netstat -n grep 8080 | grep TIME_WAIT P.S: Just restarted the servers. As expected the thread counts dropped to normal. Now I am on 43 Current and 20 Busy. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 11:05 AM, Mick Knutson mknut...@baselogic.com wrote: I am on something right now like this. While running, Do a: netstat -n grep [your port number] | grep CALL_WAIT See if you are hung in CALL+WAIT status. --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 4:01 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: 24
Re: Tomcat current thread count
Oops. You are correct. I have been up for 2 weeks on this and need to sleep. But that was my issue. --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 5:45 AM, Rainer Jung rainer.j...@kippdata.dewrote: On 25.06.2010 11:22, Mick Knutson wrote: TIME_WAIT is fine. Not an issue. That just means they are ready to take requests. CALL_WAIT is not good, if they stick around. If this is an OOM error, then have you started this server and attached javaVisualVM onto it to see what the threads and memory are doing? Not judging on whether that's the right approach, but you probably meant CLOSE_WAIT. There is no TCP state CALL_WAIT. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat current thread count
From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Im actually using Javamelody to monitor the apps. Yet still nothing. Any ideas ? Looks like you have two separate problems. The first is that your Tomcat-configured threads get stuck somewhere, so Tomcat keeps creating more until it hits the specified limit. Looking at a thread dump should show you where they are stuck so you can fix your webapp (or database) and correct that issue. The second problem is that the JVM blew up, possibly due to your webapp creating even more threads outside of the control of Tomcat, or perhaps eating up some other resource that provokes an out of memory condition. Again, a thread dump before the JVM dies should give you an idea of what's going on. All indications are that you have a poorly implemented webapp that's not managing resources properly. Also, you should always provide the exact Tomcat version you're using, and the platform you're running on. The only version information you provided (and that by accident) is the JVM level. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Hi, My system consists of: Rhel 5.3 Tomcat v 6.0.26 Java(TM) SE Runtime Environment (build 1.6.0_16-b01) I do not see anything else in the catalina.out or any other log file apart from the one I have posted before. What would be the possible ways of finding where the problem lies ? Thanks Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 3:58 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Im actually using Javamelody to monitor the apps. Yet still nothing. Any ideas ? Looks like you have two separate problems. The first is that your Tomcat-configured threads get stuck somewhere, so Tomcat keeps creating more until it hits the specified limit. Looking at a thread dump should show you where they are stuck so you can fix your webapp (or database) and correct that issue. The second problem is that the JVM blew up, possibly due to your webapp creating even more threads outside of the control of Tomcat, or perhaps eating up some other resource that provokes an out of memory condition. Again, a thread dump before the JVM dies should give you an idea of what's going on. All indications are that you have a poorly implemented webapp that's not managing resources properly. Also, you should always provide the exact Tomcat version you're using, and the platform you're running on. The only version information you provided (and that by accident) is the JVM level. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Can you run jvisualvm.exe against it? --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 10:31 AM, Ozgur Ozdemircili ozgur.ozdemirc...@gmail.com wrote: Hi, My system consists of: Rhel 5.3 Tomcat v 6.0.26 Java(TM) SE Runtime Environment (build 1.6.0_16-b01) I do not see anything else in the catalina.out or any other log file apart from the one I have posted before. What would be the possible ways of finding where the problem lies ? Thanks Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 3:58 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count Im actually using Javamelody to monitor the apps. Yet still nothing. Any ideas ? Looks like you have two separate problems. The first is that your Tomcat-configured threads get stuck somewhere, so Tomcat keeps creating more until it hits the specified limit. Looking at a thread dump should show you where they are stuck so you can fix your webapp (or database) and correct that issue. The second problem is that the JVM blew up, possibly due to your webapp creating even more threads outside of the control of Tomcat, or perhaps eating up some other resource that provokes an out of memory condition. Again, a thread dump before the JVM dies should give you an idea of what's going on. All indications are that you have a poorly implemented webapp that's not managing resources properly. Also, you should always provide the exact Tomcat version you're using, and the platform you're running on. The only version information you provided (and that by accident) is the JVM level. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat current thread count
From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count What would be the possible ways of finding where the problem lies ? As previously stated, take a thread dump *before* the JVM crashes and find out what your threads are stuck on. Your monitoring tool probably may have that capability built-in; if it doesn't, look here: http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
You can force it by: set JAVA_OPTS=%JAVA_OPTS% -*XX:+HeapDumpOnCtrlBreak*-XX:+HeapDumpOnOutOfMemoryError --- Thank You… Mick Knutson, President BASE Logic, Inc. Enterprise Architecture, Design, Mentoring Agile Consulting p. (866) BLiNC-411: (254-6241-1) f. (415) 685-4233 Website: http://www.baselogic.com Blog: http://www.baselogic.com/blog/ Linked IN: http://linkedin.com/in/mickknutson Twitter: http://twitter.com/mickknutson Vacation Rental: http://tahoe.baselogic.com --- On Fri, Jun 25, 2010 at 10:56 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count What would be the possible ways of finding where the problem lies ? As previously stated, take a thread dump *before* the JVM crashes and find out what your threads are stuck on. Your monitoring tool probably may have that capability built-in; if it doesn't, look here: http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat current thread count
From: Mick Knutson [mailto:mknut...@baselogic.com] Subject: Re: Tomcat current thread count You can force it by: set JAVA_OPTS=%JAVA_OPTS% -*XX:+HeapDumpOnCtrlBreak*-XX:+HeapDumpOnOutOfMemoryError No, that gets you a Java heap dump, not a thread dump - not of interest here. The out-of-memory error that brought down the JVM was for the C heap of the process space, not the Java heap. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Hi, Thanks for the useful link. I have gotten a thread dump using kill -3 . You can find it in the attached file. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 4:56 PM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count What would be the possible ways of finding where the problem lies ? As previously stated, take a thread dump *before* the JVM crashes and find out what your threads are stuck on. Your monitoring tool probably may have that capability built-in; if it doesn't, look here: http://wiki.apache.org/tomcat/HowTo#How_do_I_obtain_a_thread_dump_of_my_running_webapp_.3F - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org 2010-06-25 17:11:59 Full thread dump Java HotSpot(TM) Server VM (14.2-b01 mixed mode): Thread-74 daemon prio=10 tid=0x0a971c00 nid=0x35a6 waiting on condition [0x6ab51000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.serhs.agents.mikitravel.robots.JAXBPoolTimer.run(JAXBPoolTimer.java:37) Thread-73 daemon prio=10 tid=0x0a4e1000 nid=0x35a5 waiting on condition [0x6abf2000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.serhs.agents.mikitravel.robots.MappingCacheTimer.run(MappingCacheTimer.java:41) http-8080-6 daemon prio=10 tid=0x0a0a6000 nid=0x34f2 in Object.wait() [0x6aa0c000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x8b779e08 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) - locked 0x8b779e08 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) at java.lang.Thread.run(Thread.java:619) http-8080-5 daemon prio=10 tid=0x09fc2000 nid=0x4e79 in Object.wait() [0x6aba1000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x8b346d78 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) - locked 0x8b346d78 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) at java.lang.Thread.run(Thread.java:619) http-8080-4 daemon prio=10 tid=0x0a7ed400 nid=0x4e77 in Object.wait() [0x6aa5d000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x8b348f10 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) - locked 0x8b348f10 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) at java.lang.Thread.run(Thread.java:619) http-8080-3 daemon prio=10 tid=0x0a7ebc00 nid=0x4e76 in Object.wait() [0x6aaae000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x8b348f90 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at java.lang.Object.wait(Object.java:485) at org.apache.tomcat.util.net.JIoEndpoint$Worker.await(JIoEndpoint.java:458) - locked 0x8b348f90 (a org.apache.tomcat.util.net.JIoEndpoint$Worker) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:484) at java.lang.Thread.run(Thread.java:619) Timer-9 daemon prio=10 tid=0x0a117800 nid=0x468a in Object.wait() [0x6ac43000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x8ad8ef20 (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) - locked 0x8ad8ef20 (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) Timer-8 daemon prio=10 tid=0x0a5b3400 nid=0x45e1 in Object.wait() [0x6ac94000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) - waiting on 0x76360980
Re: Tomcat current thread count
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Özgür, On 6/25/2010 11:39 AM, Ozgur Ozdemircili wrote: Thanks for the useful link. I have gotten a thread dump using kill -3 . You can find it in the attached file. It looks like almost everything is idle, here. Can you wait a while, when you estimate you might be close to encountering this error, and then take another thread dump? That one would probably be more interesting. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwk0H4ACgkQ9CaO5/Lv0PBn8wCeO/6Hot7d6IUsmhR960kwh8EN BZYAn14PHLLP4vdUDUP6OQ0EXyLiZmj5 =KX94 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
On 25.06.2010 17:51, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Özgür, On 6/25/2010 11:39 AM, Ozgur Ozdemircili wrote: Thanks for the useful link. I have gotten a thread dump using kill -3 . You can find it in the attached file. It looks like almost everything is idle, here. Can you wait a while, when you estimate you might be close to encountering this error, and then take another thread dump? That one would probably be more interesting. Yup, or maybe write a small shell script, sending the signal every 5 minutes or maybe even once every minute in order to make it likely to catch the problematic status. Be prepared for enough space in the logs directory though. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
Hi, I probably wont be able to get the one close to crash but the jstack.out in the attached file contains the output while the applicaction is receiving requests. Hope it will be more helpful. Özgür Özdemircili http://www.acikkod.org Code so clean you could eat off it On Fri, Jun 25, 2010 at 5:51 PM, Christopher Schultz ch...@christopherschultz.net wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Özgür, On 6/25/2010 11:39 AM, Ozgur Ozdemircili wrote: Thanks for the useful link. I have gotten a thread dump using kill -3 . You can find it in the attached file. It looks like almost everything is idle, here. Can you wait a while, when you estimate you might be close to encountering this error, and then take another thread dump? That one would probably be more interesting. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwk0H4ACgkQ9CaO5/Lv0PBn8wCeO/6Hot7d6IUsmhR960kwh8EN BZYAn14PHLLP4vdUDUP6OQ0EXyLiZmj5 =KX94 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat current thread count
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Özgür, On 6/25/2010 12:21 PM, Ozgur Ozdemircili wrote: I probably wont be able to get the one close to crash but the jstack.out in the attached file contains the output while the applicaction is receiving requests. Hope it will be more helpful. The list stripped your attachment. Try giving it a .txt extension. Sounds stupid, but it appears that works sometimes. I'm guessing that not much help will come from a thread dump during some normal processing... you'll just see your code running normally. What usually trips-up threads is deadlock or long-running requests piling up. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkwk758ACgkQ9CaO5/Lv0PBnUACfW2XW2+jZc93diac3BrOGzVWh sQQAnjILLLv0n1lX4lFtvN8FPg0cbABy =jm31 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Tomcat current thread count
From: Ozgur Ozdemircili [mailto:ozgur.ozdemirc...@gmail.com] Subject: Re: Tomcat current thread count I have gotten a thread dump using kill -3 . Don't see anything wrong with the Tomcat threads. As Chris noted, they are all just waiting for another request to show up, which is normal. However, you do have several Timer-x threads going, which makes me suspicious that you webapp is starting up extra threads and not properly managing them. Simply making them daemon (as they are) is not sufficient, since the daemon threads won't terminate until Tomcat is fully shut down. There are also some other threads running that start in your webapp, including one waiting on a UDP socket. This still looks like poor resource management by your webapp. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
