Tomcat Hangs with variable frequency

2010-12-22 Thread Rhonny David
Hi ALL,



We are using tomcat 6.0 in production environment and facing a critical problem 
that tomcat hangs with variable frequency in a week. We have tried to figure 
out 
the problem but couldn't reach the root cause. To investigate the cause we used 
the lambda probe tool. Lamba probe shows multiple threads are blocked when 
tomcat hangs. Normally i have seen a similar type of stack trace for the thread 
which lambda probe shows as blocked, I am pasting it as follows:

net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 ) 
net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 ) 
net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 ) 
net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 ) 
net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 ) 
net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 ) 
net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close ( 
DelegatingResultSet.java:187 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close ( 
DelegatingStatement.java:163 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate ( 
DelegatingConnection.java:426 ) 
org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close ( 
DelegatingConnection.java:246 ) 
org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose ( 
PoolableConnection.java:122 ) 
org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject ( 
PoolableConnectionFactory.java:628 ) 
org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject ( 
GenericObjectPool.java:1247 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject ( 
AbandonedObjectPool.java:125 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned ( 
AbandonedObjectPool.java:158 ) 
org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject ( 
AbandonedObjectPool.java:77 ) 
org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection ( 
PoolingDataSource.java:106 ) 
org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection ( 
BasicDataSource.java:1044 ) 
com.avanza.contactcenter.dbutil.DBConnection.getConnection ( 
DBConnection.java:154 ) 
com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList
 ( TableFieldsMappingProcessor.java:696 ) 
com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch
 ( CustomerDetailActionProcessor.java:187 ) 
com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo
 ( RdvInboundActionProcessor.java:817 ) 
com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage
 ( RendezvousMessageSource.java:431 ) 
com.avanza.contactcenter.integration.RendezvousMessageSource.run ( 
RendezvousMessageSource.java:136 )


As we have been investigating the issue from a month, so i would like to share 
it with you though you can understand the scenario in depth. Initially we 
thought that there are connection leak problems in our application because the 
lambda probe showed us that db connections are continuously busy. We scanned 
 twice the DAO layer of our application to search any connection leaks, but we 
didn't find any. At that time our application's Context.xml  file didn't 
contain 
the removeAbandoned property. After DAO layer scan, we decide to set 
the removeAbandoned = true, so that connection pool refreshes at run time. But 
this also didn't work for us. Lambda probe still showed us blocked threads when 
tomcat hanged, but yes by setting the removeAbandoned = true connection pool 
continued to refresh.

When tomcat hanged today we noticed following statistics on lambda probe, 
(along 
with blocked threads for which stack trace is already provided above):



System OVerView

Memory utilization
 -- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB 


OS information
JVM:  Java(TM) 2 Runtime Environment, Standard Edition  1.5.0_10-b03  Java 
HotSpot(TM) Client VM) 
OS: Windows 2003 (Service Pack 2) x86 5.2
Processors: 8
Current time: Wed Dec 22 13:00:48 GMT+05:00 2010
Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0


Current memory usage

 Used Committed   MaximumInitial  Group
 
Survivor Space:   1.26Mb   4.63Mb  7.88Mb 4.56MbHEAP 

Perm Gen:   34.28Mb  34.50Mb 64.00Mb8.00Mb   NON_HEAP 
  
Tenured Gen:  115.20Mb  553.88Mb  945.25Mb  553.88Mb   HEAP 


Eden Space: 2.08Mb   37.00Mb63.00Mb37.00Mb   HEAP 


Code Cache12.37Mb  12.47Mb  32.00Mb  192.00Kb  NON_HEAP 


Total165.65Mb  642.47Mb  1.09Gb  603.63Mb  TOTAL 


OS information
OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free 
RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb 

(In above line,Free RAM is 2GB, it might be due to that operating

Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Mark Thomas
On 22/12/2010 11:48, Rhonny David wrote:
 Hi ALL,
 
 
 
 We are using tomcat 6.0 in production environment

Exactly which version? Also exact versions for JDK and OS please.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Rhonny David


I think i have already provided these except tomcat. Anyways we are using 
Tomcat 
6.0.29, which is pointed to jre 1.5.0_10. And the OS is Microsoft Windows 
Server 
2003, R2 , Standard Edition, Service Pack2.

Regards,
David




From: Mark Thomas ma...@apache.org
To: Tomcat Users List users@tomcat.apache.org
Sent: Wed, December 22, 2010 4:52:52 PM
Subject: Re: Tomcat Hangs with variable frequency

On 22/12/2010 11:48, Rhonny David wrote:
 Hi ALL,
 
 
 
 We are using tomcat 6.0 in production environment

Exactly which version? Also exact versions for JDK and OS please.

Mark

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


  

Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Konstantin Kolinko
 net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )

I think it is an issue with implementation of JtdsResultSet.close(), [1]

0485:if (!getConnection().isClosed()) {
0486:// Skip to end of result set
0487:// Could send cancel but this is safer as
0488:// cancel could kill other statements
in a batch.
0489:while (next())
0490:;
0491:}

[1] 
http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm

[2] 
http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467

It tries to dispose the resultset by reading through it and hangs in
the process (or maybe the resultset is too big and reading it takes a
while).

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Rhonny David
I won't support your reasoning, because the same JTDS driver version we are 
using for our other site and there the number of concurrent users of the 
application is more than 800 and therefore the volume of concurrent requests is 
very high, but we never faced the issue of this nature. Secondly, we have 
recently started using JTD driver and tomcat jndi connection pooling for our 
current site and earlier we used microsoft driver and our own connection 
pooling 
mechaninsm, but the same issue of tomcat hanging was occurring at that time.


Regards,
David




From: Konstantin Kolinko knst.koli...@gmail.com
To: Tomcat Users List users@tomcat.apache.org
Sent: Wed, December 22, 2010 6:53:46 PM
Subject: Re: Tomcat Hangs with variable frequency

 net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )

I think it is an issue with implementation of JtdsResultSet.close(), [1]

0485:if (!getConnection().isClosed()) {
0486:// Skip to end of result set
0487:// Could send cancel but this is safer as
0488:// cancel could kill other statements
in a batch.
0489:while (next())
0490:;
0491:}

[1] 
http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm


[2] 
http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467


It tries to dispose the resultset by reading through it and hangs in
the process (or maybe the resultset is too big and reading it takes a
while).

Best regards,
Konstantin Kolinko

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org


  

Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Konstantin Kolinko
If closing the result set requires reading it in full (as JTDS does),
it can take significant time.

You should take thread dumps to see what the threads are doing and
what they are waiting for. (Three thread dumps taken several seconds
apart).

Post them inline, or somewhere else (like pastebin site). Attachments
are usually stripped off by the mailing list software.

2010/12/22 Rhonny David rhonnyda...@yahoo.com:
 I won't support your reasoning, because the same JTDS driver version we are
 using for our other site and there the number of concurrent users of the
 application is more than 800 and therefore the volume of concurrent requests 
 is
 very high, but we never faced the issue of this nature. Secondly, we have
 recently started using JTD driver and tomcat jndi connection pooling for our
 current site and earlier we used microsoft driver and our own connection 
 pooling
 mechaninsm, but the same issue of tomcat hanging was occurring at that time.


 Regards,
 David



 
 From: Konstantin Kolinko knst.koli...@gmail.com
 To: Tomcat Users List users@tomcat.apache.org
 Sent: Wed, December 22, 2010 6:53:46 PM
 Subject: Re: Tomcat Hangs with variable frequency

 net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 )

 I think it is an issue with implementation of JtdsResultSet.close(), [1]

 0485:                        if (!getConnection().isClosed()) {
 0486:                            // Skip to end of result set
 0487:                            // Could send cancel but this is safer as
 0488:                            // cancel could kill other statements
 in a batch.
 0489:                            while (next())
 0490:                                ;
 0491:                        }

 [1]
 http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm


 [2]
 http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467


 It tries to dispose the resultset by reading through it and hangs in
 the process (or maybe the resultset is too big and reading it takes a
 while).

 Best regards,
 Konstantin Kolinko



-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org



Re: Tomcat Hangs with variable frequency

2010-12-22 Thread Christopher Schultz
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Konstantin,

On 12/22/2010 11:56 AM, Konstantin Kolinko wrote:
 If closing the result set requires reading it in full (as JTDS does),
 it can take significant time.

Especially if the queries are something like

SELECT * FROM huge_table

 You should take thread dumps to see what the threads are doing and
 what they are waiting for. (Three thread dumps taken several seconds
 apart).

I would also inspect the queries that are in progress when these
connections appear to hang.

We had a problem with Oracle long ago where the table indexes had just
become horribly fragmented and performance slowed to a crawl. In an act
of desperation, someone ran OPTIMIZE TABLE on one of the tables
involved in particularly slow queries and basically everything got
better. From then on out, we called that turning off the 'suck' bit on
Oracle. :)

I would definitely like to see what queries are running on the server
and what their status is.

The problem is certainly /not/ Tomcat, here: threads are stuck in a
3rd-party library connecting to a 3rd-party database. Tomcat just
happens to be in the stack trace because that's the DBCP in use: the
driver appears to be hanging, not Tomcat's DBCP.

- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk0SNXUACgkQ9CaO5/Lv0PDAUQCfWgdgZY61qly2Z0xDP4vhZcbX
OyYAn14lv4chUAmiD4h4z3jqgDPxOGda
=Fqs8
-END PGP SIGNATURE-

-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org