Re: VM thread dump from windows service
perfect. this works nicely! thanks, patrick On 4/22/05, Anoop kumar V [EMAIL PROTECTED] wrote: Patrick, I am not sure if this will help you but it is worth checking out Stack trace at the following link: http://tmitevski.users.mcs2.netarray.com/stacktrace/app/launch.jnlp let me know if it helped.. -Anoop On 4/22/05, Patrick Lacson [EMAIL PROTECTED] wrote: If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Thanks and best regards, Anoop -- Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: VM thread dump from windows service
Spoke too soon.. :( Our tomcat is installed using the tomcat.exe service installer that ships with the tomcat 4.1.x series. This Stack Trace tool cannot do a thread dump on the process id of tomcat.exe -Patrick On 4/25/05, Patrick Lacson [EMAIL PROTECTED] wrote: perfect. this works nicely! thanks, patrick On 4/22/05, Anoop kumar V [EMAIL PROTECTED] wrote: Patrick, I am not sure if this will help you but it is worth checking out Stack trace at the following link: http://tmitevski.users.mcs2.netarray.com/stacktrace/app/launch.jnlp let me know if it helped.. -Anoop On 4/22/05, Patrick Lacson [EMAIL PROTECTED] wrote: If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Thanks and best regards, Anoop -- Patrick -- Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: VM thread dump from windows service
Hey, I am using the Java Service Wrapper from http://wrapper.tanukisoftware.org/doc/english/introduction.html. The Service Wrapper have a JMX Mbean that have a stack dump operation. http://wrapper.tanukisoftware.org/doc/english/jmx.html Send me a direct mail to get a LifecycleListener that register the Mbean Peter Patrick Lacson schrieb: Spoke too soon.. :( Our tomcat is installed using the tomcat.exe service installer that ships with the tomcat 4.1.x series. This Stack Trace tool cannot do a thread dump on the process id of tomcat.exe -Patrick On 4/25/05, Patrick Lacson [EMAIL PROTECTED] wrote: perfect. this works nicely! thanks, patrick On 4/22/05, Anoop kumar V [EMAIL PROTECTED] wrote: Patrick, I am not sure if this will help you but it is worth checking out Stack trace at the following link: http://tmitevski.users.mcs2.netarray.com/stacktrace/app/launch.jnlp let me know if it helped.. -Anoop On 4/22/05, Patrick Lacson [EMAIL PROTECTED] wrote: If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Thanks and best regards, Anoop -- Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: VM thread dump from windows service
This looks promising. the application detects VM instances and can pull a thread-dump on that instance.. i'll try it on monday and see how it goes. thanks! On 4/22/05, Anoop kumar V [EMAIL PROTECTED] wrote: Patrick, I am not sure if this will help you but it is worth checking out Stack trace at the following link: http://tmitevski.users.mcs2.netarray.com/stacktrace/app/launch.jnlp let me know if it helped.. -Anoop On 4/22/05, Patrick Lacson [EMAIL PROTECTED] wrote: If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Thanks and best regards, Anoop -- Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
VM thread dump from windows service
If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: VM thread dump from windows service
Patrick, I am not sure if this will help you but it is worth checking out Stack trace at the following link: http://tmitevski.users.mcs2.netarray.com/stacktrace/app/launch.jnlp let me know if it helped.. -Anoop On 4/22/05, Patrick Lacson [EMAIL PROTECTED] wrote: If I run tomcat as a windows service, what is the best way to get a VM thread dump? From a command window it's pretty easy with the Control-Break command.. what are my options for doing the equivalent from a running tomcat windows service? Thanks, Patrick - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- Thanks and best regards, Anoop - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Thread dump for Windows Service
Hello Robert, Yes it exists a realy good way to got controlled Threadumps. Install your Tomcat with Java Service Wrapper and activate the Wrapper JMX Bean with a operation to create a threaddump and reload the complete server. Links: Wrapper http://wrapper.tanukisoftware.org/doc/english/index.html http://wrapper.tanukisoftware.org/doc/english/jmx.html Centaurus Platform ( Full featured Tomcat Hosting Bundle) http://centaurus.sourceforge.net/ My Windows Example: http://tomcat.objektpark.org/examples/04_09_tomcat_example.zip regards Peter Robert Herold schrieb: I've installed tomcat as a Windows service. Is there any way to trigger the JVM to produce a thread dump? I understand how to do so when Tomcat is run in a console window (cntl-break), but in this case it has to be run as a service. Thanks for any pointers... -- bob -- Robert Herold [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- J2EE Systemarchitekt und Tomcat Experte http://objektpark.de/ http://www.webapp.de/ Am Josephsschacht 72, 44879 Bochum, Deutschland Telefon: (49) 234 9413228 Mobil:(49) 175 1660884 E-Mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Thread dump for Windows Service
I've installed tomcat as a Windows service. Is there any way to trigger the JVM to produce a thread dump? I understand how to do so when Tomcat is run in a console window (cntl-break), but in this case it has to be run as a service. Thanks for any pointers... -- bob -- Robert Herold [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Broken Pipe (continued) - Thread Dump Results
Hola, I did a thread dump (SIG-QUIT) on both the JRun and Tomcat environments during the refresh button load test and the differences are significant. The differences in internal thread management between any two servers are likely to be significant. Tomcat: Java CPU% exceeds 80%. Site becomes unusable. Thread dump yields approximately 70 threads, just over half are waiting for the DB. Maybe a connection pool problem? I realize this test and its methodology are somewhat arbitrary, but can ANYONE explain what Tomcat (with mod_jk or mod_jk2) is doing with its thread management with excessive broken pipes and why the threads pile up out of control? Can you provide a webapp and test script for us to reproduce your results? PS. I presumed it was bad form to attach documents, but I do have thread dumps available if anyone is willing to take a gander. Not necessarily bad form, but the list won't allow most attachment types. Snip the relevant parts of the tomcat thread dump (the JRun is useless, as the servers are completely different) and post it as a normal message, not an attachment. Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Broken Pipe (continued) - Thread Dump Results
Our server does recover (whether JRun or Tomcat), but as Pepijn points out, the thread management during the behaviour is the main concern. I did a thread dump (SIG-QUIT) on both the JRun and Tomcat environments during the refresh button load test and the differences are significant. Between the two environments I observed the following: Tomcat: Java CPU% exceeds 80%. Site becomes unusable. Thread dump yields approximately 70 threads, just over half are waiting for the DB. JRun: Java reaches approximately 42% CPU. The site is usable and improves over time. Thread dump yields only about 18 threads, mostly in Object.wait() state. I realize this test and its methodology are somewhat arbitrary, but can ANYONE explain what Tomcat (with mod_jk or mod_jk2) is doing with its thread management with excessive broken pipes and why the threads pile up out of control? PS. I presumed it was bad form to attach documents, but I do have thread dumps available if anyone is willing to take a gander. -- Chris Boyce - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
I definitely do have LD_KERNEL_ASSUME in my environment If you search for LD_KERNEL and my name on google, you will see that I was part of a discussion a while back that was glad to find that out. Daniel Gibby Oscar Carrillo wrote: Hi, Saw your message on the boards. Did you make sure you have this environment variable set? On systems that I need it, I put it in my tomcat startup file. LD_KERNEL_ASSUME=2.2.5 Check out my howto page for my brief notes on threading and it's potential problems with JVMs and threads. http://www.linuxjava.net/howto/webapp/ Oscar On Tue, 17 Feb 2004, Daniel Gibby wrote: My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. The machine has a relatively light load. I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 I've fixed problems in the past on a separate java application (not tomcat) where I can tell what the problem is in the thread dump because a thread waiting to be notified is also the one that has a lock on it to notify the thing that is waiting to notify it... (that didn't make sense, I know... but anyway it is basically a circle where it won't ever get woken up.) However, in this tomcat case, I can't see anything like that where something is waiting in circles... even though I wouldn't rule that out. My experience on reading thread dumps is limited... Anyway, can someone who has better experience tell me what is consuming the CPU? Restarting tomcat brings the load back down, and it slowly goes up again... like over a few days to a weeks time it is back up to 8.00 Load Average. I won't include the whole file. I trimmed the file down to 1350 lines by getting rid of a lot of 2HPMEMMAPLINE lines and the section titled: 0SECTION CL subcomponent dump routine but I think that is still too long to post here. I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Thanks, Daniel NULL 0SECTION TITLE subcomponent dump routine NULL === 1TISIGINFO signal 3 received 1TIDATETIMEDate: 2004/02/17 at 08:53:22 1TIFILENAMEJavacore filename:/tmp/javacore.20040217.085322.23429.txt NULL 0SECTION XHPI subcomponent dump routine NULL == 1HPTIMETue Feb 17 08:53:22 2004 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??. 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930 NULL 1HPOPENV Operating Environment NULL - 2HPHOSTNAMEHost : somehost.com.(none) 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002 2HPLIBCVER glibc Version: 2.2.93 2HPCPUSProcessors - 3HPARCH Architecture : (not implemented) 3HPNUMCPUS How Many : (not implemented) 3HPCPUSENABLED Enabled : 1 NULL 1HPMEMINFO Memory Info NULL --- 2HPMEMLINE total:used:free: shared: buffers: cached: 2HPMEMLINE Mem: 1055625216 1015181312 404439040 83464192 614227968 2HPMEMLINE Swap: 1052827648 929792 1051897856 2HPMEMLINE MemTotal: 1030884 kB 2HPMEMLINE MemFree: 39496 kB 2HPMEMLINE MemShared: 0 kB 2HPMEMLINE Buffers: 81508 kB 2HPMEMLINE Cached: 599596 kB 2HPMEMLINE SwapCached:236 kB 2HPMEMLINE Active: 552968 kB 2HPMEMLINE Inact_dirty:344020 kB 2HPMEMLINE Inact_clean: 50304 kB 2HPMEMLINE Inact_target: 189456 kB 2HPMEMLINE HighTotal: 130880 kB 2HPMEMLINE HighFree: 1024 kB 2HPMEMLINE LowTotal: 94 kB 2HPMEMLINE LowFree: 38472 kB 2HPMEMLINE SwapTotal: 1028152 kB 2HPMEMLINE SwapFree: 1027244 kB 2HPMEMLINE Committed_AS: 1067972 kB NULL 1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) - NULL --- 2HPUSERLIMIT RLIMIT_FSIZE : infinity 2HPUSERLIMIT RLIMIT_DATA: infinity 2HPUSERLIMIT RLIMIT_STACK : 2093056 2HPUSERLIMIT RLIMIT_CORE: 0 2HPUSERLIMIT RLIMIT_NOFILE : 1024 2HPUSERLIMIT RLIMIT_NPROC : 7168 NULL 1HPSIGHANDLERS JVM Signal Handlers NULL --- 2HPSIGHANDLER HUP: unknown handler 2HPSIGHANDLER INT: unknown handler 2HPSIGHANDLER QUIT
Re: thread dump analysis
Hi, Saw your message on the boards. Did you make sure you have this environment variable set? On systems that I need it, I put it in my tomcat startup file. LD_KERNEL_ASSUME=2.2.5 Check out my howto page for my brief notes on threading and it's potential problems with JVMs and threads. http://www.linuxjava.net/howto/webapp/ Oscar On Tue, 17 Feb 2004, Daniel Gibby wrote: My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. The machine has a relatively light load. I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 I've fixed problems in the past on a separate java application (not tomcat) where I can tell what the problem is in the thread dump because a thread waiting to be notified is also the one that has a lock on it to notify the thing that is waiting to notify it... (that didn't make sense, I know... but anyway it is basically a circle where it won't ever get woken up.) However, in this tomcat case, I can't see anything like that where something is waiting in circles... even though I wouldn't rule that out. My experience on reading thread dumps is limited... Anyway, can someone who has better experience tell me what is consuming the CPU? Restarting tomcat brings the load back down, and it slowly goes up again... like over a few days to a weeks time it is back up to 8.00 Load Average. I won't include the whole file. I trimmed the file down to 1350 lines by getting rid of a lot of 2HPMEMMAPLINE lines and the section titled: 0SECTION CL subcomponent dump routine but I think that is still too long to post here. I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Thanks, Daniel NULL 0SECTION TITLE subcomponent dump routine NULL === 1TISIGINFO signal 3 received 1TIDATETIMEDate: 2004/02/17 at 08:53:22 1TIFILENAMEJavacore filename:/tmp/javacore.20040217.085322.23429.txt NULL 0SECTION XHPI subcomponent dump routine NULL == 1HPTIMETue Feb 17 08:53:22 2004 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??. 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930 NULL 1HPOPENV Operating Environment NULL - 2HPHOSTNAMEHost : somehost.com.(none) 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002 2HPLIBCVER glibc Version: 2.2.93 2HPCPUSProcessors - 3HPARCH Architecture : (not implemented) 3HPNUMCPUS How Many : (not implemented) 3HPCPUSENABLED Enabled : 1 NULL 1HPMEMINFO Memory Info NULL --- 2HPMEMLINE total:used:free: shared: buffers: cached: 2HPMEMLINE Mem: 1055625216 1015181312 404439040 83464192 614227968 2HPMEMLINE Swap: 1052827648 929792 1051897856 2HPMEMLINE MemTotal: 1030884 kB 2HPMEMLINE MemFree: 39496 kB 2HPMEMLINE MemShared: 0 kB 2HPMEMLINE Buffers: 81508 kB 2HPMEMLINE Cached: 599596 kB 2HPMEMLINE SwapCached:236 kB 2HPMEMLINE Active: 552968 kB 2HPMEMLINE Inact_dirty:344020 kB 2HPMEMLINE Inact_clean: 50304 kB 2HPMEMLINE Inact_target: 189456 kB 2HPMEMLINE HighTotal: 130880 kB 2HPMEMLINE HighFree: 1024 kB 2HPMEMLINE LowTotal: 94 kB 2HPMEMLINE LowFree: 38472 kB 2HPMEMLINE SwapTotal: 1028152 kB 2HPMEMLINE SwapFree: 1027244 kB 2HPMEMLINE Committed_AS: 1067972 kB NULL 1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) - NULL --- 2HPUSERLIMIT RLIMIT_FSIZE : infinity 2HPUSERLIMIT RLIMIT_DATA: infinity 2HPUSERLIMIT RLIMIT_STACK : 2093056 2HPUSERLIMIT RLIMIT_CORE: 0 2HPUSERLIMIT RLIMIT_NOFILE : 1024 2HPUSERLIMIT RLIMIT_NPROC : 7168 NULL 1HPSIGHANDLERS JVM Signal Handlers NULL --- 2HPSIGHANDLER HUP: unknown handler 2HPSIGHANDLER INT: unknown handler 2HPSIGHANDLER QUIT : unknown handler 2HPSIGHANDLER ILL: unknown handler 2HPSIGHANDLER TRAP : unknown handler 2HPSIGHANDLER
thread dump analysis
My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. The machine has a relatively light load. I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 I've fixed problems in the past on a separate java application (not tomcat) where I can tell what the problem is in the thread dump because a thread waiting to be notified is also the one that has a lock on it to notify the thing that is waiting to notify it... (that didn't make sense, I know... but anyway it is basically a circle where it won't ever get woken up.) However, in this tomcat case, I can't see anything like that where something is waiting in circles... even though I wouldn't rule that out. My experience on reading thread dumps is limited... Anyway, can someone who has better experience tell me what is consuming the CPU? Restarting tomcat brings the load back down, and it slowly goes up again... like over a few days to a weeks time it is back up to 8.00 Load Average. I won't include the whole file. I trimmed the file down to 1350 lines by getting rid of a lot of 2HPMEMMAPLINE lines and the section titled: 0SECTION CL subcomponent dump routine but I think that is still too long to post here. I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Thanks, Daniel NULL 0SECTION TITLE subcomponent dump routine NULL === 1TISIGINFO signal 3 received 1TIDATETIMEDate: 2004/02/17 at 08:53:22 1TIFILENAMEJavacore filename:/tmp/javacore.20040217.085322.23429.txt NULL 0SECTION XHPI subcomponent dump routine NULL == 1HPTIMETue Feb 17 08:53:22 2004 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??. 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930 NULL 1HPOPENV Operating Environment NULL - 2HPHOSTNAMEHost : somehost.com.(none) 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002 2HPLIBCVER glibc Version: 2.2.93 2HPCPUSProcessors - 3HPARCH Architecture : (not implemented) 3HPNUMCPUS How Many : (not implemented) 3HPCPUSENABLED Enabled : 1 NULL 1HPMEMINFO Memory Info NULL --- 2HPMEMLINE total:used:free: shared: buffers: cached: 2HPMEMLINE Mem: 1055625216 1015181312 404439040 83464192 614227968 2HPMEMLINE Swap: 1052827648 929792 1051897856 2HPMEMLINE MemTotal: 1030884 kB 2HPMEMLINE MemFree: 39496 kB 2HPMEMLINE MemShared: 0 kB 2HPMEMLINE Buffers: 81508 kB 2HPMEMLINE Cached: 599596 kB 2HPMEMLINE SwapCached:236 kB 2HPMEMLINE Active: 552968 kB 2HPMEMLINE Inact_dirty:344020 kB 2HPMEMLINE Inact_clean: 50304 kB 2HPMEMLINE Inact_target: 189456 kB 2HPMEMLINE HighTotal: 130880 kB 2HPMEMLINE HighFree: 1024 kB 2HPMEMLINE LowTotal: 94 kB 2HPMEMLINE LowFree: 38472 kB 2HPMEMLINE SwapTotal: 1028152 kB 2HPMEMLINE SwapFree: 1027244 kB 2HPMEMLINE Committed_AS: 1067972 kB NULL 1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) - NULL --- 2HPUSERLIMIT RLIMIT_FSIZE : infinity 2HPUSERLIMIT RLIMIT_DATA: infinity 2HPUSERLIMIT RLIMIT_STACK : 2093056 2HPUSERLIMIT RLIMIT_CORE: 0 2HPUSERLIMIT RLIMIT_NOFILE : 1024 2HPUSERLIMIT RLIMIT_NPROC : 7168 NULL 1HPSIGHANDLERS JVM Signal Handlers NULL --- 2HPSIGHANDLER HUP: unknown handler 2HPSIGHANDLER INT: unknown handler 2HPSIGHANDLER QUIT : unknown handler 2HPSIGHANDLER ILL: unknown handler 2HPSIGHANDLER TRAP : unknown handler 2HPSIGHANDLER ABRT : unknown handler 2HPSIGHANDLER FPE: unknown handler 2HPSIGHANDLER KILL : default handler - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: thread dump analysis
Howdy, Can you reproduce this behavior when running with a profiler? Yoav Shapira Millennium ChemInformatics -Original Message- From: Daniel Gibby [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 17, 2004 12:21 PM To: Tomcat Users List Subject: thread dump analysis My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. The machine has a relatively light load. I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 I've fixed problems in the past on a separate java application (not tomcat) where I can tell what the problem is in the thread dump because a thread waiting to be notified is also the one that has a lock on it to notify the thing that is waiting to notify it... (that didn't make sense, I know... but anyway it is basically a circle where it won't ever get woken up.) However, in this tomcat case, I can't see anything like that where something is waiting in circles... even though I wouldn't rule that out. My experience on reading thread dumps is limited... Anyway, can someone who has better experience tell me what is consuming the CPU? Restarting tomcat brings the load back down, and it slowly goes up again... like over a few days to a weeks time it is back up to 8.00 Load Average. I won't include the whole file. I trimmed the file down to 1350 lines by getting rid of a lot of 2HPMEMMAPLINE lines and the section titled: 0SECTION CL subcomponent dump routine but I think that is still too long to post here. I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Thanks, Daniel NULL --- - 0SECTION TITLE subcomponent dump routine NULL === 1TISIGINFO signal 3 received 1TIDATETIMEDate: 2004/02/17 at 08:53:22 1TIFILENAMEJavacore filename: /tmp/javacore.20040217.085322.23429.txt NULL --- - 0SECTION XHPI subcomponent dump routine NULL == 1HPTIMETue Feb 17 08:53:22 2004 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??. 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930 NULL 1HPOPENV Operating Environment NULL - 2HPHOSTNAMEHost : somehost.com.(none) 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002 2HPLIBCVER glibc Version: 2.2.93 2HPCPUSProcessors - 3HPARCH Architecture : (not implemented) 3HPNUMCPUS How Many : (not implemented) 3HPCPUSENABLED Enabled : 1 NULL 1HPMEMINFO Memory Info NULL --- 2HPMEMLINE total:used:free: shared: buffers: cached: 2HPMEMLINE Mem: 1055625216 1015181312 404439040 83464192 614227968 2HPMEMLINE Swap: 1052827648 929792 1051897856 2HPMEMLINE MemTotal: 1030884 kB 2HPMEMLINE MemFree: 39496 kB 2HPMEMLINE MemShared: 0 kB 2HPMEMLINE Buffers: 81508 kB 2HPMEMLINE Cached: 599596 kB 2HPMEMLINE SwapCached:236 kB 2HPMEMLINE Active: 552968 kB 2HPMEMLINE Inact_dirty:344020 kB 2HPMEMLINE Inact_clean: 50304 kB 2HPMEMLINE Inact_target: 189456 kB 2HPMEMLINE HighTotal: 130880 kB 2HPMEMLINE HighFree: 1024 kB 2HPMEMLINE LowTotal: 94 kB 2HPMEMLINE LowFree: 38472 kB 2HPMEMLINE SwapTotal: 1028152 kB 2HPMEMLINE SwapFree: 1027244 kB 2HPMEMLINE Committed_AS: 1067972 kB NULL 1HPUSERLIMITS User Limits (in bytes except for NOFILE and NPROC) - NULL --- 2HPUSERLIMIT RLIMIT_FSIZE : infinity 2HPUSERLIMIT RLIMIT_DATA: infinity 2HPUSERLIMIT RLIMIT_STACK : 2093056 2HPUSERLIMIT RLIMIT_CORE: 0 2HPUSERLIMIT RLIMIT_NOFILE : 1024 2HPUSERLIMIT RLIMIT_NPROC : 7168 NULL 1HPSIGHANDLERS JVM Signal Handlers NULL --- 2HPSIGHANDLER HUP: unknown handler 2HPSIGHANDLER INT: unknown handler 2HPSIGHANDLER QUIT : unknown handler 2HPSIGHANDLER ILL: unknown handler 2HPSIGHANDLER TRAP : unknown handler 2HPSIGHANDLER ABRT : unknown handler 2HPSIGHANDLER FPE: unknown handler 2HPSIGHANDLER KILL : default handler - To unsubscribe, e-mail: [EMAIL PROTECTED
Re: thread dump analysis
I can't run a profiler on this as it is a live system. I also don't have the resources to setup another box right now to even test it... I'm in a fire-fighting mode half of the time, and a village building mode the rest of the time. (Hmmm. I think I'll stop everything and write a tech book called Village Building and Fire Fighting - Stopping the Cycle of Technology Wastefulness - Don't anyone take my title! This list is archived and I can prove that I thought of it first! I will make a multi player board game as well. People will be able to play it at work during meetings!) Anyway, back to the problem at hand. I can't run a profiler right now. It is too big of an obstacle right now, even though I'd like to be able to take three weeks and run a profile on all my java apps. I just don't have three weeks, and the barrier to entry is too big since I'm basically a one-man show. Thanks, Daniel == All original works from this email Copyright 2004 Daniel Gibby - Steal and be Sued! - or contact me for licensing! == Shapira, Yoav wrote: Howdy, Can you reproduce this behavior when running with a profiler? Yoav Shapira Millennium ChemInformatics -Original Message- From: Daniel Gibby [mailto:[EMAIL PROTECTED] Sent: Tuesday, February 17, 2004 12:21 PM To: Tomcat Users List Subject: thread dump analysis My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. The machine has a relatively light load. I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 I've fixed problems in the past on a separate java application (not tomcat) where I can tell what the problem is in the thread dump because a thread waiting to be notified is also the one that has a lock on it to notify the thing that is waiting to notify it... (that didn't make sense, I know... but anyway it is basically a circle where it won't ever get woken up.) However, in this tomcat case, I can't see anything like that where something is waiting in circles... even though I wouldn't rule that out. My experience on reading thread dumps is limited... Anyway, can someone who has better experience tell me what is consuming the CPU? Restarting tomcat brings the load back down, and it slowly goes up again... like over a few days to a weeks time it is back up to 8.00 Load Average. I won't include the whole file. I trimmed the file down to 1350 lines by getting rid of a lot of 2HPMEMMAPLINE lines and the section titled: 0SECTION CL subcomponent dump routine but I think that is still too long to post here. I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Thanks, Daniel NULL --- - 0SECTION TITLE subcomponent dump routine NULL === 1TISIGINFO signal 3 received 1TIDATETIMEDate: 2004/02/17 at 08:53:22 1TIFILENAMEJavacore filename: /tmp/javacore.20040217.085322.23429.txt NULL --- - 0SECTION XHPI subcomponent dump routine NULL == 1HPTIMETue Feb 17 08:53:22 2004 1HPSIGRECV SIGQUIT received in ?? at (nil) in ??. 1HPFULLVERSION J2RE 1.4.1 IBM build cxia321411-20030930 NULL 1HPOPENV Operating Environment NULL - 2HPHOSTNAMEHost : somehost.com.(none) 2HPOSLEVEL OS Level : 2.4.18-14.#1 Wed Sep 4 13:35:50 EDT 2002 2HPLIBCVER glibc Version: 2.2.93 2HPCPUSProcessors - 3HPARCH Architecture : (not implemented) 3HPNUMCPUS How Many : (not implemented) 3HPCPUSENABLED Enabled : 1 NULL 1HPMEMINFO Memory Info NULL --- 2HPMEMLINE total:used:free: shared: buffers: cached: 2HPMEMLINE Mem: 1055625216 1015181312 404439040 83464192 614227968 2HPMEMLINE Swap: 1052827648 929792 1051897856 2HPMEMLINE MemTotal: 1030884 kB 2HPMEMLINE MemFree: 39496 kB 2HPMEMLINE MemShared: 0 kB 2HPMEMLINE Buffers: 81508 kB 2HPMEMLINE Cached: 599596 kB 2HPMEMLINE SwapCached:236 kB 2HPMEMLINE Active: 552968 kB 2HPMEMLINE Inact_dirty:344020 kB 2HPMEMLINE Inact_clean: 50304 kB 2HPMEMLINE Inact_target
RE: thread dump analysis
Howdy, My tomcat 4.1.29 instance running J2RE 1.4.1 IBM build cxia321411-20030930 on RedHat 9 kernel 2.4.18-14 keeps gaining processor usage until finally can't answer requests successfully. OK, so you can't profile. I'm not sure a thread dump would be that useful in this scenario because it might be a repeating operation, e.g. on a growing collection. Do you maybe have a class that keeps searching through a collection for some object, and that collection grows with time to the point where the search consumes a lot of CPU? Or perhaps garbage collection is the culprit? If you can restart your JVM, add -verbose:gc to the java VM options, so you can see if the CPU time is spent collecting garbage is monotonically increasing to the point of thrashing (repeating full GC collections that don't reclaim any significant heap space). Yoav Shapira This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
On Tue, February 17, 2004 at 9:20 am, Daniel Gibby wrote: I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 snip I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Best to just post the entire dump somewhere on the web where everyone can look at it. Sounds like something somewhere is getting threads stuck in a loop, should be pretty easy to spot if you get the dump during a non-busy time. Not sure how big of an attachment the list will accept, but you might be able to compress it and attach it to the list. If it's too big, just send it to me privately and I will have a look. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
Well, I'd rather not show the world what my java processes are doing in case there is something proprietary in there. I'll send it to you personally. Daniel David Rees wrote: On Tue, February 17, 2004 at 9:20 am, Daniel Gibby wrote: I did a kill -3 on the process that showed up on top and got a stack trace... the problem is I have no idea how to analyze the thread dump to see what is consuming CPU. I'm sure something must be spinning its wheels, but I don't know how to tell... I can just see that when I run top my tomcat process has 99.9 % of the CPU and the load average is 8.00 8.00 8.00 snip I'm hoping that someone can tell me what to include and what to exclude and I'll reply with the appropriate parts of the dump. Best to just post the entire dump somewhere on the web where everyone can look at it. Sounds like something somewhere is getting threads stuck in a loop, should be pretty easy to spot if you get the dump during a non-busy time. Not sure how big of an attachment the list will accept, but you might be able to compress it and attach it to the list. If it's too big, just send it to me privately and I will have a look. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote: Well, I'd rather not show the world what my java processes are doing in case there is something proprietary in there. I'll send it to you personally. OK, but it's tough for people to help troubleshoot your issue unless you do so. ;-) -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
David Rees wrote, On 2/17/2004 12:43 PM: On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote: Well, I'd rather not show the world what my java processes are doing in case there is something proprietary in there. I'll send it to you personally. OK, but it's tough for people to help troubleshoot your issue unless you do so. ;-) I had a look at Daniel's thread dump, it did not appear to be an issue with Tomcat, but rather an issue with either some jcrontab threads or a JVM/OS issue as he is running the IBM 1.4.1 JDK on a stock RH 8 system. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: thread dump analysis
I'm going to try and disable the jcrontab servlets and see if the problem persists. After that, I'll try and schedule some upgrades. Thanks for your help David! Daniel David Rees wrote: David Rees wrote, On 2/17/2004 12:43 PM: On Tue, February 17, 2004 1at 2:04 pm, Daniel Gibby wrote: Well, I'd rather not show the world what my java processes are doing in case there is something proprietary in there. I'll send it to you personally. OK, but it's tough for people to help troubleshoot your issue unless you do so. ;-) I had a look at Daniel's thread dump, it did not appear to be an issue with Tomcat, but rather an issue with either some jcrontab threads or a JVM/OS issue as he is running the IBM 1.4.1 JDK on a stock RH 8 system. -Dave - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: Need help to study thread dump
Howdy, It's a classic thread safety issue. Dump your homebrew connection pooling implementation in favor of a 3rd party one like DBCP that is better tested against these race conditions. Yoav Shapira Millennium ChemInformatics -Original Message- From: Vikas [mailto:[EMAIL PROTECTED] Sent: Friday, December 12, 2003 7:56 PM To: [EMAIL PROTECTED] Subject: Need help to study thread dump Hi, My app is running on bes 5.1 App sometime stops responsding and i have to restart my server. Below is some portion of thread dump which i got for the app VBJ ThreadPool Worker daemon prio=5 tid=0x14f1148 nid=0xad70 waiting for monitor entry [e767d000..e767fc24] at com.vks.group.pkg.util.DBConnectionPool.getConnection(DBConnectionPool. java :166) - waiting to lock efdc0158 (a com.vks.group.pkg.util.DBConnectionPool) at com.vks.group.pkg.util.DBConnectionPool.isDBPoolOK(DBConnectionPool.jav a:34 9) at com.vks.group.pkg.util.DBConnectionPoolManager.isDBPoolOK(DBConnectionP oolM anager.java:219) at com.vks.group.pkg.util.DBUtil.isDBPoolOK(DBUtil.java:681) at com.vks.group.pkg.util.DBUtil.isDBPoolOK(DBUtil.java:677) at MyServResponse.processRequest(MyServResponse.java:39) From the dump all i could make out is there lock wait on one object to get database connection. Is it problem with the java code or like problem with the database unavailabilty. We dont get this problem quiet frequently , this come only once in a month , ie we need to start our server once every month. Has anybody come across such problem in past , pls help me how to go about solving it. ** Viks ** Yahoo! India Matrimony: Find your partner online.Post your profile. This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Need help to study thread dump
Hi, My app is running on bes 5.1 App sometime stops responsding and i have to restart my server. Below is some portion of thread dump which i got for the app VBJ ThreadPool Worker daemon prio=5 tid=0x14f1148 nid=0xad70 waiting for monitor entry [e767d000..e767fc24] at com.vks.group.pkg.util.DBConnectionPool.getConnection(DBConnectionPool.java:166) - waiting to lock efdc0158 (a com.vks.group.pkg.util.DBConnectionPool) at com.vks.group.pkg.util.DBConnectionPool.isDBPoolOK(DBConnectionPool.java:349) at com.vks.group.pkg.util.DBConnectionPoolManager.isDBPoolOK(DBConnectionPoolManager.java:219) at com.vks.group.pkg.util.DBUtil.isDBPoolOK(DBUtil.java:681) at com.vks.group.pkg.util.DBUtil.isDBPoolOK(DBUtil.java:677) at MyServResponse.processRequest(MyServResponse.java:39) From the dump all i could make out is there lock wait on one object to get database connection. Is it problem with the java code or like problem with the database unavailabilty. We dont get this problem quiet frequently , this come only once in a month , ie we need to start our server once every month. Has anybody come across such problem in past , pls help me how to go about solving it. ** Viks ** Yahoo! India Matrimony: Find your partner online.Post your profile.
[OT] How to get a thread dump (was: Severe notification ...)
Mufaddal, Is there a facility in tomcat that would allow me to dump its threads and allow me to see where the threads are waiting and not moving on ? People need to stop asking this question and do google java virtual machine thread dump. Sheesh... Send the process a STOP signal, and it will emit a thread dump to standard output. This is CTRL-\ on UNIX within the terminal that started the Java process (if it's still in the foreground). You can go a CTRL-BREAK on win32 to get the same thing. I highly recommend redirecting stdout to a file, as you'll get LOTS of output from the thread dump. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
RE: [OT] How to get a thread dump (was: Severe notification ...)
Howdy, At least on unix, it's a QUIT (not STOP) signal. kill -s QUIT processId Where processId, is, well, the process ID of your JVM ;) Yoav Shapira Millennium ChemInformatics -Original Message- From: Christopher Schultz [mailto:[EMAIL PROTECTED] Sent: Wednesday, November 05, 2003 8:27 AM To: Tomcat Users List Subject: [OT] How to get a thread dump (was: Severe notification ...) Mufaddal, Is there a facility in tomcat that would allow me to dump its threads and allow me to see where the threads are waiting and not moving on ? People need to stop asking this question and do google java virtual machine thread dump. Sheesh... Send the process a STOP signal, and it will emit a thread dump to standard output. This is CTRL-\ on UNIX within the terminal that started the Java process (if it's still in the foreground). You can go a CTRL-BREAK on win32 to get the same thing. I highly recommend redirecting stdout to a file, as you'll get LOTS of output from the thread dump. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] This e-mail, including any attachments, is a confidential business communication, and may contain information that is confidential, proprietary and/or privileged. This e-mail is intended only for the individual(s) to whom it is addressed, and may not be saved, copied, printed, disclosed or used by anyone else. If you are not the(an) intended recipient, please immediately delete this e-mail from your computer system and notify the sender. Thank you. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT] How to get a thread dump (was: Severe notification ...)
Yoav, Howdy, At least on unix, it's a QUIT (not STOP) signal. kill -s QUIT processId Oh! Brain fart. Yep, it's QUIT, not STOP. It's still CTRL-\ from the terminal. -chris - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Thread dump - (basic)
What is the best method to get a stack dump following a frozen tomcat? I cannot use kill -QUIT (pid) since running in production and do not want to run TC in the foreground. Thanks, Euan J2DSK_1.4.1_03 TC 4.1.24 (Standalone, Coyote connector) Linux RH9. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: Thread dump - (basic)
Hi, Kill -QUIT will work. It will send the output to catalina.out and not kill the process. It doesn't have to be running in the foreground. -e On Mon, 1 Sep 2003, Euan Guttridge wrote: What is the best method to get a stack dump following a frozen tomcat? I cannot use kill -QUIT (pid) since running in production and do not want to run TC in the foreground. Thanks, Euan J2DSK_1.4.1_03 TC 4.1.24 (Standalone, Coyote connector) Linux RH9. - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Thread dump
I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: Thread dump
kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
Beg your pardon? would that not actually kill the process, rather than displaying the thread dump? And what if one wants to see the thread dump right from the moment tomcat starts up? Thanks, manav. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:18 PM To: Tomcat Users List Subject: Re: Thread dump kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
You could use Thread.enumerate(Thread [] ) and then do a Thread.dumpStack() on each thread in your code. I can't provide any definite answers on how to get a thread dump right from the moment Tomcat starts, but I suppose you could modify Tomcat code (call the about the methods in your code) to do this. RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 10:01 AMSubject: RE: Thread dump Please respond to Tomcat Users List Beg your pardon? would that not actually kill the process, rather than displaying the thread dump? And what if one wants to see the thread dump right from the moment tomcat starts up? Thanks, manav. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:18 PM To: Tomcat Users List Subject: Re: Thread dump kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Re: Thread dump
Run it under a JPDA debugger and use something like - Xrunjdwp:transport=dt_shmem,server=y,suspend=y so that the JVM stops and waits for debugger commands right away. Then you could get to the right point in your code, and with any decent debugger look at the thread dump without needing to send the signal. By the way, what exactly do you mean by right from the moment tomcat starts up? Do you mean before the tomcat's main() starts? Before your servlet is loaded? dwh [EMAIL PROTECTED] wrote: You could use Thread.enumerate(Thread [] ) and then do a Thread.dumpStack() on each thread in your code. I can't provide any definite answers on how to get a thread dump right from the moment Tomcat starts, but I suppose you could modify Tomcat code (call the about the methods in your code) to do this. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
Thanks. I'm gonna try jpda - i have just started exploring it. i was interested to see the thread dump from the point tomcat's main() starts, to get a hang of the sequence of the class loaders/resource managers/security realms, etc starting up. thanks, manav. -Original Message- From: Denis Haskin [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:53 PM To: Tomcat Users List Subject: Re: Thread dump Run it under a JPDA debugger and use something like - Xrunjdwp:transport=dt_shmem,server=y,suspend=y so that the JVM stops and waits for debugger commands right away. Then you could get to the right point in your code, and with any decent debugger look at the thread dump without needing to send the signal. By the way, what exactly do you mean by right from the moment tomcat starts up? Do you mean before the tomcat's main() starts? Before your servlet is loaded? dwh [EMAIL PROTECTED] wrote: You could use Thread.enumerate(Thread [] ) and then do a Thread.dumpStack() on each thread in your code. I can't provide any definite answers on how to get a thread dump right from the moment Tomcat starts, but I suppose you could modify Tomcat code (call the about the methods in your code) to do this. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
FYI, no it doesn't, it just causes the (Sun, at least) JVM to dump a list of threads and their stacks to stderr. Note that it's the real stderr, not System.err. This is a JVM feature. It can be done anytime and is a *really* useful debugging feature. Jeff -Original Message- From: Manavendra Gupta [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 8:01 AM To: Tomcat Users List Subject: RE: Thread dump Beg your pardon? would that not actually kill the process, rather than displaying the thread dump? And what if one wants to see the thread dump right from the moment tomcat starts up? Thanks, manav. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:18 PM To: Tomcat Users List Subject: Re: Thread dump kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
I am intrigued by this feature; it would help with the debugging of a application. I tried to test it against tomcat but I get nothing on stderr (i.e., nothing in /var/log/messages, terminal, directory I am in, catalina.out, or any of the logs for Tomcat). Could you expand on what behavior you see when you send the -3 to Tomcat's PID? Thanks, Ben Ricker Wellinx.com On Fri, 2002-12-06 at 12:29, Schnitzer, Jeff wrote: FYI, no it doesn't, it just causes the (Sun, at least) JVM to dump a list of threads and their stacks to stderr. Note that it's the real stderr, not System.err. This is a JVM feature. It can be done anytime and is a *really* useful debugging feature. Jeff -Original Message- From: Manavendra Gupta [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 8:01 AM To: Tomcat Users List Subject: RE: Thread dump Beg your pardon? would that not actually kill the process, rather than displaying the thread dump? And what if one wants to see the thread dump right from the moment tomcat starts up? Thanks, manav. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:18 PM To: Tomcat Users List Subject: Re: Thread dump kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- Ben Ricker [EMAIL PROTECTED] Wellinx.com -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
RE: Thread dump
There's a whitepaper about it located here: http://developer.java.sun.com/developer/technicalArticles/Programming/Stackt race/ Jeremy -Original Message- From: Ben Ricker [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 12:58 PM To: Tomcat Users List Subject: RE: Thread dump I am intrigued by this feature; it would help with the debugging of a application. I tried to test it against tomcat but I get nothing on stderr (i.e., nothing in /var/log/messages, terminal, directory I am in, catalina.out, or any of the logs for Tomcat). Could you expand on what behavior you see when you send the -3 to Tomcat's PID? Thanks, Ben Ricker Wellinx.com On Fri, 2002-12-06 at 12:29, Schnitzer, Jeff wrote: FYI, no it doesn't, it just causes the (Sun, at least) JVM to dump a list of threads and their stacks to stderr. Note that it's the real stderr, not System.err. This is a JVM feature. It can be done anytime and is a *really* useful debugging feature. Jeff -Original Message- From: Manavendra Gupta [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 8:01 AM To: Tomcat Users List Subject: RE: Thread dump Beg your pardon? would that not actually kill the process, rather than displaying the thread dump? And what if one wants to see the thread dump right from the moment tomcat starts up? Thanks, manav. -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] Sent: Friday, December 06, 2002 9:18 PM To: Tomcat Users List Subject: Re: Thread dump kill -3 pid RS Manavendra Gupta To: Tomcat Users List [EMAIL PROTECTED] [EMAIL PROTECTED] cc: 12/06/02 09:53 AMSubject: Thread dump Please respond to Tomcat Users List I have tomcat 4.1 running on Linux. How do i see the thread dump? The startup.sh on linux just starts it in the background, while i could use startup.bat on windows and get the thread dump. thanks, manav. -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] -- Ben Ricker [EMAIL PROTECTED] Wellinx.com -- To unsubscribe, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] For additional commands, e-mail: mailto:tomcat-user- [EMAIL PROTECTED] -- To unsubscribe, e-mail: mailto:[EMAIL PROTECTED] For additional commands, e-mail: mailto:[EMAIL PROTECTED]
Tomcat still falling over. Thread dump help ?
After 3-4 months my tomcat continues to fall falt on it's face every now and then. Currentl it's running at 2-3 times a day ! Every time the processor usage jumps to 100% jaken up by the Java virtual machine. Then the java process jsut dies with no indication of why. This time I've got a dump using CTRL-Break. Any ieads from this why the process is behaving like this? This is Win2000, Tomcat 3.2.2 and apache 1.3.12 ANy sugesstions please, help put me out of my misery ! Andy C Full thread dump: Thread-33 (TID:0xb4a2c0, sys_thread_t:0x1e16e88, Win32ID:0x798, state:CW) prio=5 org.apache.tomcat.util.ThreadPool$MonitorRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-32 (TID:0xb4a3b0, sys_thread_t:0x1e16da0, Win32ID:0x69c, state:MW) prio=5 java.lang.Object.wait(Compiled Code) Thread-31 (TID:0xb4a360, sys_thread_t:0x1e16cb8, Win32ID:0x690, state:R) p rio=5 java.net.URL.init(Compiled Code) java.net.URL.init(Compiled Code) java.lang.ClassLoader.getSystemResource(Compiled Code) org.apache.tomcat.loader.AdaptiveClassLoader.getResource(Compiled Code) org.apache.jasper.servlet.JasperLoader.getResource(Compiled Code) org.apache.jasper.servlet.JasperLoader.getResourceAsStream(Compiled Code ) java.beans.Beans.instantiate(Compiled Code) _0002findex_0002ejsp_0002findex_jsp_1._jspService(Compiled Code) org.apache.jasper.runtime.HttpJspBase.service(Compiled Code) javax.servlet.http.HttpServlet.service(Compiled Code) org.apache.jasper.servlet.JspServlet$JspServletWrapper.service(Compiled Code) org.apache.jasper.servlet.JspServlet.serviceJspFile(Compiled Code) org.apache.jasper.servlet.JspServlet.service(Compiled Code) javax.servlet.http.HttpServlet.service(Compiled Code) org.apache.tomcat.core.ServletWrapper.doService(Compiled Code) org.apache.tomcat.core.Handler.service(Compiled Code) org.apache.tomcat.core.ServletWrapper.service(Compiled Code) org.apache.tomcat.core.ContextManager.internalService(Compiled Code) org.apache.tomcat.core.ContextManager.service(Compiled Code) org.apache.tomcat.service.connector.Ajp12ConnectionHandler.processConnec tion(Compiled Code) Thread-30 (TID:0xb4a2f0, sys_thread_t:0x1e16bd0, Win32ID:0x67c, state:MW) prio=5 org.apache.tomcat.util.ThreadPool$ControlRunnable.runIt(Compiled Code) Thread-29 (TID:0xb4a5e8, sys_thread_t:0x1e16ae8, Win32ID:0x698, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-28 (TID:0xb4a5a8, sys_thread_t:0x1e16a00, Win32ID:0x580, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-27 (TID:0xb4a550, sys_thread_t:0x1e16918, Win32ID:0x5b8, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-26 (TID:0xb4a510, sys_thread_t:0x1e16830, Win32ID:0x2c8, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-25 (TID:0xb4a4c8, sys_thread_t:0x1e02630, Win32ID:0x5ac, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-24 (TID:0xb4a600, sys_thread_t:0x1e02548, Win32ID:0x5dc, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-23 (TID:0xb4aab8, sys_thread_t:0x1e02460, Win32ID:0x6b8, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-22 (TID:0xb4aa50, sys_thread_t:0x1e02378, Win32ID:0x574, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-21 (TID:0xb4aa10, sys_thread_t:0x1e02290, Win32ID:0x45c, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-20 (TID:0xb4a9d0, sys_thread_t:0x1e021a8, Win32ID:0x55c, state:CW) prio=5 java.lang.Object.wait(Compiled Code) org.apache.tomcat.util.ThreadPool$ControlRunnable.run(Compiled Code) java.lang.Thread.run(Compiled Code) Thread-19 (TID:0xb4a988
