Re: tomcat 6.0.29 hung

2010-10-26 Thread Jason Britton
I've further traced the root cause of my problems with threads sitting on socket read, never timing out, never throwing an exception and sometimes causing application deadlock (depending on where the socket read occurs) down to the network. By comparing the output of lsof -i :1521 | grep on

Re: tomcat 6.0.29 hung

2010-10-22 Thread Jason Britton
So while working on reproducing tomcat webapp hang as earlier discussed, a different tomcat serving up a single webapp on same box also stopped responding, jvm for this tomcat is still running but when trying to reach it via browser I get error page of unable to connect. Thread dumps for both

RE: tomcat 6.0.29 hung

2010-10-22 Thread Caldarale, Charles R
From: Jason Britton [mailto:jbritto...@gmail.com] Subject: Re: tomcat 6.0.29 hung My thoughts are this is the thread causing all the backup AJP Thread 1 RUNNABLE, sitting on socketRead0, Has locked 0x2aaad303cad0 (a com.acompany.model.customer.CustomerData) which almost all other

Re: tomcat 6.0.29 hung

2010-10-22 Thread Jason Britton
) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:619) On Fri, Oct 22, 2010 at 6:34 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Jason Britton [mailto:jbritto...@gmail.com] Subject: Re: tomcat 6.0.29 hung My thoughts

Re: tomcat 6.0.29 hung

2010-10-22 Thread Pid
:908) at java.lang.Thread.run(Thread.java:619) On Fri, Oct 22, 2010 at 6:34 AM, Caldarale, Charles R chuck.caldar...@unisys.com wrote: From: Jason Britton [mailto:jbritto...@gmail.com] Subject: Re: tomcat 6.0.29 hung My thoughts are this is the thread causing all the backup

Re: tomcat 6.0.29 hung

2010-10-22 Thread Jason Britton
one of two of the hanging applications are doing db connection validation. The application using DBCP configures it's SharedPoolDataSource using setValidationQuery(select 'validationQuery' from dual); setTestOnBorrow(true); Funny enough, thread stacktrace below shows

Re: tomcat 6.0.29 hung

2010-10-22 Thread Dave Mansfield
Jason Britton wrote: So while working on reproducing tomcat webapp hang as earlier discussed, a different tomcat serving up a single webapp on same box also stopped responding, jvm for this tomcat is still running but when trying to reach it via browser I get error page of unable to connect.

Re: tomcat 6.0.29 hung

2010-10-22 Thread Jason Britton
Thanks for the insight Dave. I have several threads sitting spinning on socket read at the moment and dba reports zero database locks of any kind on database they're trying to read from (a qa database). Quite baffled as to what's happening to these database connections and why every now and then

Re: tomcat 6.0.29 hung

2010-10-22 Thread Dave Mansfield
This might be related, and don't know if you've read it - it's a bit out of date, and depends on versions of stuff you're using, Jason. http://forums.sun.com/thread.jspa?threadID=343023start=15tstart=0 -- D Mansfield Jason Britton wrote: Thanks for the insight Dave. I have several threads

Re: tomcat 6.0.29 hung

2010-10-22 Thread Pid
On 22/10/2010 16:47, Jason Britton wrote: one of two of the hanging applications are doing db connection validation. The application using DBCP configures it's SharedPoolDataSource using setValidationQuery(select 'validationQuery' from dual); That looks weird. Is that really

RE: tomcat 6.0.29 hung

2010-10-22 Thread Caldarale, Charles R
From: Dave Mansfield [mailto:dmansfi...@sbcglobal.net] Subject: Re: tomcat 6.0.29 hung This might be related, and don't know if you've read it - it's a bit out of date, and depends on versions of stuff you're using, Jason. http://forums.sun.com/thread.jspa?threadID=343023start=15tstart

[OT] VisualVM, (was Re: tomcat 6.0.29 hung)

2010-10-21 Thread Pid
On 21/10/2010 00:11, Jason Britton wrote: Highly recommend trying out jvisualvm (it's in your jdk bin) if you're running 1.6. There's a standalone release - 1.3.1 - with more plugins some slightly shinier toys. https://visualvm.dev.java.net/ Even if you can't run it on the same machine as

Re: [OT] VisualVM, (was Re: tomcat 6.0.29 hung)

2010-10-21 Thread Jason Britton
I'll definitely will take a look at the standalone release. Thanks for the info. I'm going to throw Cygwin/X on my desktop machine to hopefully be able to pull up jvisualvm remotely and/or just use standalone version. Perhaps Cygwin/X would also allow you to pull up VisualGC remotely? On Thu,

Re: tomcat 6.0.29 hung

2010-10-21 Thread Martin Kuen
jason, Out of personal interest I have a question about the way you use jstack. I came across this jstack cannot take dump of hung process problem before. I once witnessed the same exceptions when chasing a deadlock. This happened to me on ubuntu 10.4.0 (and 10.4.1) 64-bit using sun's jdk

Re: tomcat 6.0.29 hung

2010-10-21 Thread Wesley Acheson
On Thu, Oct 21, 2010 at 1:11 AM, Jason Britton jbritto...@gmail.com wrote: Thankfully when I pulled up jvisualvm on the server and issued thread dumps, even though the stacktraces for the threads did not come up within jvisualvm the thread stacktraces were dumped to stdout. I think this is

Re: [OT] VisualVM, (was Re: tomcat 6.0.29 hung)

2010-10-21 Thread Pid
On 21/10/2010 13:49, Jason Britton wrote: I'll definitely will take a look at the standalone release. Thanks for the info. I'm going to throw Cygwin/X on my desktop machine to hopefully be able to pull up jvisualvm remotely and/or just use standalone version. Perhaps Cygwin/X would also

Re: tomcat 6.0.29 hung

2010-10-21 Thread Mark Thomas
On 20/10/2010 20:20, Jason Britton wrote: commons-dbcp-1.2.1.jar and commons-pool-1.3.jar Lots of sync related fixes since then. I'd strongly recommend and upgrade. Mark - To unsubscribe, e-mail:

Re: tomcat 6.0.29 hung

2010-10-21 Thread Jason Britton
Hi Martin, In researching my jstack not working issue I too came across the make sure you run jstack as the same user that tomcat is running as. So I did try su'ing to my tomcat user and issuing the jstack -F -l -m pid command to no avail. After I reproduce my application deadlock again this

Re: tomcat 6.0.29 hung

2010-10-21 Thread André Warnier
Jason Britton wrote: Hi Martin, In researching my jstack not working issue I too came across the make sure you run jstack as the same user that tomcat is running as. So I did try su'ing to my tomcat user and issuing the jstack -F -l -m pid command to no avail. After I reproduce my application

tomcat 6.0.29 hung

2010-10-20 Thread Jason Britton
I have a tomcat instance hanging up (and currently still hung up) and would really appreciate pointers on how to debug further. A belated thanks to Konstantin for the info on kill -QUIT pid to retrieve thread dumps But I was only able to pull thread dumps on the hung process using jvisualvm.

Re: tomcat 6.0.29 hung

2010-10-20 Thread Pid
On 20/10/2010 21:37, Jason Britton wrote: I have a tomcat instance hanging up (and currently still hung up) and would really appreciate pointers on how to debug further. A belated thanks to Konstantin for the info on kill -QUIT pid to retrieve thread dumps But I was only able to pull thread

Re: tomcat 6.0.29 hung

2010-10-20 Thread Phil Steitz
On 10/20/10 7:11 PM, Jason Britton wrote: Thankfully when I pulled up jvisualvm on the server and issued thread dumps, even though the stacktraces for the threads did not come up within jvisualvm the thread stacktraces were dumped to stdout. So I did get thread stacktraces in my tomcat log.

Re: tomcat 6.0.29 hung

2010-10-20 Thread Jason Britton
commons-dbcp-1.2.1.jar and commons-pool-1.3.jar I have a connection handler class that I wrote around a static instance of SharedPoolDataSource. SharedPoolDataSource gets initialized in static block. Locking shown in stacktrace is occurring within aSharedPoolDataSource.getConnection() call.