Tomcat Hangs with variable frequency
Hi ALL, We are using tomcat 6.0 in production environment and facing a critical problem that tomcat hangs with variable frequency in a week. We have tried to figure out the problem but couldn't reach the root cause. To investigate the cause we used the lambda probe tool. Lamba probe shows multiple threads are blocked when tomcat hangs. Normally i have seen a similar type of stack trace for the thread which lambda probe shows as blocked, I am pasting it as follows: net.sourceforge.jtds.jdbc.SharedSocket.getNetPacket ( SharedSocket.java:693 ) net.sourceforge.jtds.jdbc.ResponseStream.getPacket ( ResponseStream.java:466 ) net.sourceforge.jtds.jdbc.ResponseStream.read ( ResponseStream.java:103 ) net.sourceforge.jtds.jdbc.TdsCore.nextToken ( TdsCore.java:2202 ) net.sourceforge.jtds.jdbc.TdsCore.getNextRow ( TdsCore.java:764 ) net.sourceforge.jtds.jdbc.JtdsResultSet.next ( JtdsResultSet.java:593 ) net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) org.apache.tomcat.dbcp.dbcp.DelegatingResultSet.close ( DelegatingResultSet.java:187 ) org.apache.tomcat.dbcp.dbcp.DelegatingStatement.close ( DelegatingStatement.java:163 ) org.apache.tomcat.dbcp.dbcp.DelegatingConnection.passivate ( DelegatingConnection.java:426 ) org.apache.tomcat.dbcp.dbcp.DelegatingConnection.close ( DelegatingConnection.java:246 ) org.apache.tomcat.dbcp.dbcp.PoolableConnection.reallyClose ( PoolableConnection.java:122 ) org.apache.tomcat.dbcp.dbcp.PoolableConnectionFactory.destroyObject ( PoolableConnectionFactory.java:628 ) org.apache.tomcat.dbcp.pool.impl.GenericObjectPool.invalidateObject ( GenericObjectPool.java:1247 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.invalidateObject ( AbandonedObjectPool.java:125 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.removeAbandoned ( AbandonedObjectPool.java:158 ) org.apache.tomcat.dbcp.dbcp.AbandonedObjectPool.borrowObject ( AbandonedObjectPool.java:77 ) org.apache.tomcat.dbcp.dbcp.PoolingDataSource.getConnection ( PoolingDataSource.java:106 ) org.apache.tomcat.dbcp.dbcp.BasicDataSource.getConnection ( BasicDataSource.java:1044 ) com.avanza.contactcenter.dbutil.DBConnection.getConnection ( DBConnection.java:154 ) com.avanza.contactcenter.db.TableFieldsMappingProcessor.getDynamicColumnsReportList ( TableFieldsMappingProcessor.java:696 ) com.avanza.contactcenter.processor.CustomerDetailActionProcessor.CustomerDetailInfoSearch ( CustomerDetailActionProcessor.java:187 ) com.avanza.contactcenter.processor.RdvInboundActionProcessor.responseCustomerDetailInfo ( RdvInboundActionProcessor.java:817 ) com.avanza.contactcenter.integration.RendezvousMessageSource.processIncomingMessage ( RendezvousMessageSource.java:431 ) com.avanza.contactcenter.integration.RendezvousMessageSource.run ( RendezvousMessageSource.java:136 ) As we have been investigating the issue from a month, so i would like to share it with you though you can understand the scenario in depth. Initially we thought that there are connection leak problems in our application because the lambda probe showed us that db connections are continuously busy. We scanned twice the DAO layer of our application to search any connection leaks, but we didn't find any. At that time our application's Context.xml file didn't contain the removeAbandoned property. After DAO layer scan, we decide to set the removeAbandoned = true, so that connection pool refreshes at run time. But this also didn't work for us. Lambda probe still showed us blocked threads when tomcat hanged, but yes by setting the removeAbandoned = true connection pool continued to refresh. When tomcat hanged today we noticed following statistics on lambda probe, (along with blocked threads for which stack trace is already provided above): System OVerView Memory utilization -- Free: 469.95 MB Total: 595.5 MB Max: 1,016.12 MB OS information JVM: Java(TM) 2 Runtime Environment, Standard Edition 1.5.0_10-b03 Java HotSpot(TM) Client VM) OS: Windows 2003 (Service Pack 2) x86 5.2 Processors: 8 Current time: Wed Dec 22 13:00:48 GMT+05:00 2010 Working dir: C:\Program Files\Apache Software Foundation\Tomcat 6.0 Current memory usage Used Committed MaximumInitial Group Survivor Space: 1.26Mb 4.63Mb 7.88Mb 4.56MbHEAP Perm Gen: 34.28Mb 34.50Mb 64.00Mb8.00Mb NON_HEAP Tenured Gen: 115.20Mb 553.88Mb 945.25Mb 553.88Mb HEAP Eden Space: 2.08Mb 37.00Mb63.00Mb37.00Mb HEAP Code Cache12.37Mb 12.47Mb 32.00Mb 192.00Kb NON_HEAP Total165.65Mb 642.47Mb 1.09Gb 603.63Mb TOTAL OS information OS Name: Windows 2003 OS Version: 5.2 Total RAM: 2.00Gb Free RAM: 2.00Gb Committed JVM memory: 681.23Mb Total swap: 4.00Gb Free swap: 4.00Gb (In above line,Free RAM is 2GB, it might be due to that operating
Re: Tomcat Hangs with variable frequency
On 22/12/2010 11:48, Rhonny David wrote: Hi ALL, We are using tomcat 6.0 in production environment Exactly which version? Also exact versions for JDK and OS please. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
I think i have already provided these except tomcat. Anyways we are using Tomcat 6.0.29, which is pointed to jre 1.5.0_10. And the OS is Microsoft Windows Server 2003, R2 , Standard Edition, Service Pack2. Regards, David From: Mark Thomas ma...@apache.org To: Tomcat Users List users@tomcat.apache.org Sent: Wed, December 22, 2010 4:52:52 PM Subject: Re: Tomcat Hangs with variable frequency On 22/12/2010 11:48, Rhonny David wrote: Hi ALL, We are using tomcat 6.0 in production environment Exactly which version? Also exact versions for JDK and OS please. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) I think it is an issue with implementation of JtdsResultSet.close(), [1] 0485:if (!getConnection().isClosed()) { 0486:// Skip to end of result set 0487:// Could send cancel but this is safer as 0488:// cancel could kill other statements in a batch. 0489:while (next()) 0490:; 0491:} [1] http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm [2] http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467 It tries to dispose the resultset by reading through it and hangs in the process (or maybe the resultset is too big and reading it takes a while). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
I won't support your reasoning, because the same JTDS driver version we are using for our other site and there the number of concurrent users of the application is more than 800 and therefore the volume of concurrent requests is very high, but we never faced the issue of this nature. Secondly, we have recently started using JTD driver and tomcat jndi connection pooling for our current site and earlier we used microsoft driver and our own connection pooling mechaninsm, but the same issue of tomcat hanging was occurring at that time. Regards, David From: Konstantin Kolinko knst.koli...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Wed, December 22, 2010 6:53:46 PM Subject: Re: Tomcat Hangs with variable frequency net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) I think it is an issue with implementation of JtdsResultSet.close(), [1] 0485:if (!getConnection().isClosed()) { 0486:// Skip to end of result set 0487:// Could send cancel but this is safer as 0488:// cancel could kill other statements in a batch. 0489:while (next()) 0490:; 0491:} [1] http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm [2] http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467 It tries to dispose the resultset by reading through it and hangs in the process (or maybe the resultset is too big and reading it takes a while). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
If closing the result set requires reading it in full (as JTDS does), it can take significant time. You should take thread dumps to see what the threads are doing and what they are waiting for. (Three thread dumps taken several seconds apart). Post them inline, or somewhere else (like pastebin site). Attachments are usually stripped off by the mailing list software. 2010/12/22 Rhonny David rhonnyda...@yahoo.com: I won't support your reasoning, because the same JTDS driver version we are using for our other site and there the number of concurrent users of the application is more than 800 and therefore the volume of concurrent requests is very high, but we never faced the issue of this nature. Secondly, we have recently started using JTD driver and tomcat jndi connection pooling for our current site and earlier we used microsoft driver and our own connection pooling mechaninsm, but the same issue of tomcat hanging was occurring at that time. Regards, David From: Konstantin Kolinko knst.koli...@gmail.com To: Tomcat Users List users@tomcat.apache.org Sent: Wed, December 22, 2010 6:53:46 PM Subject: Re: Tomcat Hangs with variable frequency net.sourceforge.jtds.jdbc.JtdsResultSet.close ( JtdsResultSet.java:486 ) I think it is an issue with implementation of JtdsResultSet.close(), [1] 0485: if (!getConnection().isClosed()) { 0486: // Skip to end of result set 0487: // Could send cancel but this is safer as 0488: // cancel could kill other statements in a batch. 0489: while (next()) 0490: ; 0491: } [1] http://www.java2s.com/Open-Source/Java-Document/Database-JDBC-Connection-Pool/jTDS/net/sourceforge/jtds/jdbc/JtdsResultSet.java.htm [2] http://jtds.svn.sourceforge.net/viewvc/jtds/branches/jTDS%201.2%20%28stable%29/src/main/net/sourceforge/jtds/jdbc/JtdsResultSet.java?revision=1107view=markup#l467 It tries to dispose the resultset by reading through it and hangs in the process (or maybe the resultset is too big and reading it takes a while). Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat Hangs with variable frequency
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Konstantin, On 12/22/2010 11:56 AM, Konstantin Kolinko wrote: If closing the result set requires reading it in full (as JTDS does), it can take significant time. Especially if the queries are something like SELECT * FROM huge_table You should take thread dumps to see what the threads are doing and what they are waiting for. (Three thread dumps taken several seconds apart). I would also inspect the queries that are in progress when these connections appear to hang. We had a problem with Oracle long ago where the table indexes had just become horribly fragmented and performance slowed to a crawl. In an act of desperation, someone ran OPTIMIZE TABLE on one of the tables involved in particularly slow queries and basically everything got better. From then on out, we called that turning off the 'suck' bit on Oracle. :) I would definitely like to see what queries are running on the server and what their status is. The problem is certainly /not/ Tomcat, here: threads are stuck in a 3rd-party library connecting to a 3rd-party database. Tomcat just happens to be in the stack trace because that's the DBCP in use: the driver appears to be hanging, not Tomcat's DBCP. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk0SNXUACgkQ9CaO5/Lv0PDAUQCfWgdgZY61qly2Z0xDP4vhZcbX OyYAn14lv4chUAmiD4h4z3jqgDPxOGda =Fqs8 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org