Re: Tomcat 6.0.35-SocketException: Too many open files issue with
On 27 Jan 2012, at 05:32, gnath gautam_exquis...@yahoo.com wrote: Hello Chris, After seeing the initial connection pool issue, i started searching online for help and i found this article : http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar. Tomcat ships with a DBCP implementing of its own. How and where are you defining the database? p Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested. However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock : Found one Java-level deadlock: = catalina-exec-1: waiting to lock monitor 0x5d7944b8 (object 0x0005bd522568, a com.mysql.jdbc.Connection), which is held by [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951] [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: waiting to lock monitor 0x5dcdea28 (object 0x0005bd659ce8, a com.mysql.jdbc.ResultSet), which is held by catalina-exec-1 Java stack information for the threads listed above: === catalina-exec-1: at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177) - waiting to lock 0x0005bd522568 (a com.mysql.jdbc.Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583) at com.mysql.jdbc.Field.getName(Field.java:487) at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593) at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926) - locked 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401) [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: at com.mysql.jdbc.ResultSet.close(ResultSet.java:736) - waiting to lock 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.Statement.realClose(Statement.java:1606) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703) at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901) - locked 0x0005bd525ba0 (a java.lang.Object) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126) at com.mysql.jdbc.Connection.realClose(Connection.java:4422) at com.mysql.jdbc.Connection.close(Connection.java:2098) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320) Please help us on this. Could it be a problem with tomcat-jdbc.jar? Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 26, 2012 9:41 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/25/12 11:53 PM, gnath wrote: As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. That will definitely do it. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same configuration properties). Why? After this change, it seemed for few hours but started seeing in the logs where the Connection Pool jar could not give any connections and seems to be all the connections are busy. So we went ahead and added a configuration property 'removeAbandoned=true' in our Datasource configuration. I would go back to DBCP
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
We defined our data sources in spring configuration file. We did not have any DB configuration defined on Server. Thanks -G From: Pid * p...@pidster.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, January 27, 2012 1:03 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with On 27 Jan 2012, at 05:32, gnath gautam_exquis...@yahoo.com wrote: Hello Chris, After seeing the initial connection pool issue, i started searching online for help and i found this article : http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar. Tomcat ships with a DBCP implementing of its own. How and where are you defining the database? p Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested. However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock : Found one Java-level deadlock: = catalina-exec-1: waiting to lock monitor 0x5d7944b8 (object 0x0005bd522568, a com.mysql.jdbc.Connection), which is held by [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951] [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: waiting to lock monitor 0x5dcdea28 (object 0x0005bd659ce8, a com.mysql.jdbc.ResultSet), which is held by catalina-exec-1 Java stack information for the threads listed above: === catalina-exec-1: at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177) - waiting to lock 0x0005bd522568 (a com.mysql.jdbc.Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583) at com.mysql.jdbc.Field.getName(Field.java:487) at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593) at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926) - locked 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401) [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: at com.mysql.jdbc.ResultSet.close(ResultSet.java:736) - waiting to lock 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.Statement.realClose(Statement.java:1606) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703) at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901) - locked 0x0005bd525ba0 (a java.lang.Object) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126) at com.mysql.jdbc.Connection.realClose(Connection.java:4422) at com.mysql.jdbc.Connection.close(Connection.java:2098) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320) Please help us on this. Could it be a problem with tomcat-jdbc.jar? Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 26, 2012 9:41 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/25/12 11:53 PM, gnath wrote: As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. That will definitely do it. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
This is top-posting, for anyone who's watching doesn't know. p On 27/01/2012 14:26, gnath wrote: We defined our data sources in spring configuration file. We did not have any DB configuration defined on Server. From: Pid * p...@pidster.com To: Tomcat Users List users@tomcat.apache.org Sent: Friday, January 27, 2012 1:03 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with On 27 Jan 2012, at 05:32, gnath gautam_exquis...@yahoo.com wrote: Hello Chris, After seeing the initial connection pool issue, i started searching online for help and i found this article : http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar. Tomcat ships with a DBCP implementing of its own. How and where are you defining the database? p Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested. However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock : Found one Java-level deadlock: = catalina-exec-1: waiting to lock monitor 0x5d7944b8 (object 0x0005bd522568, a com.mysql.jdbc.Connection), which is held by [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951] [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: waiting to lock monitor 0x5dcdea28 (object 0x0005bd659ce8, a com.mysql.jdbc.ResultSet), which is held by catalina-exec-1 Java stack information for the threads listed above: === catalina-exec-1: at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177) - waiting to lock 0x0005bd522568 (a com.mysql.jdbc.Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583) at com.mysql.jdbc.Field.getName(Field.java:487) at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593) at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926) - locked 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401) [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: at com.mysql.jdbc.ResultSet.close(ResultSet.java:736) - waiting to lock 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.Statement.realClose(Statement.java:1606) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703) at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901) - locked 0x0005bd525ba0 (a java.lang.Object) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126) at com.mysql.jdbc.Connection.realClose(Connection.java:4422) at com.mysql.jdbc.Connection.close(Connection.java:2098) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320) Please help us on this. Could it be a problem with tomcat-jdbc.jar? Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 26, 2012 9:41 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with G, On 1/25/12 11:53 PM, gnath wrote: As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
On 26/01/2012 04:53, gnath wrote: Hi Chris, Thanks a lot for looking into this and giving answers for all my questions. Sorry, i could not get chance to reply in time. As you have suggested, i started collecting the thread dumps when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same configuration properties). After this change, it seemed for few hours but started seeing in the logs where the Connection Pool jar could not give any connections and seems to be all the connections are busy. So we went ahead and added a configuration property 'removeAbandoned=true' in our Datasource configuration. We are still watching the performance and the server behavior after these changes. Will keep you posted on how things will turn out or if i see any further issues. thank you once again, I really appreciate your help. Thanks This sounds increasingly like your application isn't returning connections to the pool properly. Switching pool implementation won't help if this is the case. You should carefully examine the code where the database is used to ensure that DB resources are returned to the pool in a finally block, after use. Chris's question regarding 'what has changed' is still relevant. p From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Monday, January 23, 2012 7:51 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with G, On 1/22/12 6:18 PM, gnath wrote: We have 2 connectors (one for http and another for https) using the tomcatThreadPool. I have the connectionTimeout=2 for http connector. However i was told that our https connector might not be used by the app as our loadbalancer is handling all the https traffic and just sending them to http connector. You might want to disable that HTTPS connector, but it's probably not hurting you at all in this case -- just a bit of wasted resources. If you are sharing a thread pool then there is no negative impact on the number of threads and/or open files that you have to deal with, here. the ulimit settings were increased from default 1024 to 4096 by our admin. not sure how he did that, but i see the count as 4096 when i do ulimit -a. Well, if your admin says it's right, I suppose it's right. for ulimit -n i see its 'unlimited'. That's good. for cat /proc/PID/limits, i get the following response: Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimited seconds Max file size unlimitedunlimited bytes Max data size unlimitedunlimited bytes Max stack size10485760 unlimited bytes Max core file size0unlimited bytes Max resident set unlimitedunlimited bytes Max processes unlimitedunlimited processes Max open files4096 4096 files Max locked memory 3276832768 bytes Max address space unlimitedunlimited bytes Max file locksunlimitedunlimited locks Max pending signals 202752 202752 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Those all look good to me. This morning Tomcat hung again but this time it dint say 'too many open files' in logs but i only see this below in catalina.out: org.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Jorg.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Hmm... When it hung(java process is still up), i ran few commands like lsof by PID and couple others. Next time, take a thread dump as well. The fact that Tomcat hung up without an OS problem (like Too Many Open Files) is probably not good. If this happens again with an apparent hang with no stack traces in the logs, take a thread dump and post it back here under a different subject. here is what i got: lsof -p PID| wc -l 1342 lsof | wc -l 4520 lsof -u USER| wc -l 1953 Hmm I wonder if you are hitting a *user* or even *system* limit of some kind (though a *NIX system with a hard limit of ~4500 file descriptors seems entirely unreasonable). I also wonder how many /processes/ and/or /threads/ you have running at once. After i kill java process the lsof for pid returned obviously to zero Of course. Is there any chance that the tomcat is ignoring the ulimit
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/25/12 11:53 PM, gnath wrote: As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. That will definitely do it. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same configuration properties). Why? After this change, it seemed for few hours but started seeing in the logs where the Connection Pool jar could not give any connections and seems to be all the connections are busy. So we went ahead and added a configuration property 'removeAbandoned=true' in our Datasource configuration. I would go back to DBCP unless you think you need to switch for some reason. I suspect you are leaking database connections and don't have a suitable timeout for removal of lost database connections (or maybe didn't have that set up in the first place). You really need to enable logAbandoned so you can find out where your connection leaks are, and fix them. In development, set maxActive=1 and leave it there, forever. Also, set logAbandoned=true and always run like that in development. Running like that in production isn't a bad idea, either. We are still watching the performance and the server behavior after these changes. Will keep you posted on how things will turn out or if i see any further issues. I suspect you are still leaking connections, but your pool is now silently cleaning-up after the mess your webapp is making. Instrument your pool. Fix your leaks. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8hkDkACgkQ9CaO5/Lv0PCxFgCgs+EiV/CNjmCNekeDwKHgnNtZ 5LYAoKZUkIAJOK0eItkoHBF3wScK9lQf =AyL4 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Hello Chris, After seeing the initial connection pool issue, i started searching online for help and i found this article : http://vigilbose.blogspot.com/2009/03/apache-commons-dbcp-and-tomcat-jdbc.html so, i thought may be tomcat's jar would bring some improvement. by the way, we had commons-dbcp-1.3.jar. Do you recommend upgrading to newer commons dbcp jar instead of using tomcat-jdbc.jar. Just because we are running tomcat-6.0.35, it did not come with tomcat-jdbc.jar, so we downloaded the 1.1.1 version or jar and dropped in WEB-INF/lib and started using it. I agree what you are saying about leaking the connection and will plan to set the logAbandoned flag as you suggested. However, i was about to file a new issue but would like to describe here as well. So we have 2 servers running tomcat (same code, same configuration). After we replaced tomcat-jdbc.jar and added 'removeAbandoned' flag to true, one of the servers is doing great (ofcourse i agree that pool is cleaning up the mess), but we saw one new issue on the second server. it hasn't been releasing the connections and was consistently growing slowly. So i collected thread dump and i saw a deadlock : Found one Java-level deadlock: = catalina-exec-1: waiting to lock monitor 0x5d7944b8 (object 0x0005bd522568, a com.mysql.jdbc.Connection), which is held by [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951] [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: waiting to lock monitor 0x5dcdea28 (object 0x0005bd659ce8, a com.mysql.jdbc.ResultSet), which is held by catalina-exec-1 Java stack information for the threads listed above: === catalina-exec-1: at com.mysql.jdbc.Connection.getCharsetConverter(Connection.java:3177) - waiting to lock 0x0005bd522568 (a com.mysql.jdbc.Connection) at com.mysql.jdbc.Field.getStringFromBytes(Field.java:583) at com.mysql.jdbc.Field.getName(Field.java:487) at com.mysql.jdbc.ResultSet.buildIndexMapping(ResultSet.java:593) at com.mysql.jdbc.ResultSet.findColumn(ResultSet.java:926) - locked 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.ResultSet.getInt(ResultSet.java:2401) [Pool-Cleaner]:Tomcat Connection Pool[1-1015483951]: at com.mysql.jdbc.ResultSet.close(ResultSet.java:736) - waiting to lock 0x0005bd659ce8 (a com.mysql.jdbc.ResultSet) at com.mysql.jdbc.Statement.realClose(Statement.java:1606) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.PreparedStatement.realClose(PreparedStatement.java:1703) at com.mysql.jdbc.ServerPreparedStatement.realClose(ServerPreparedStatement.java:901) - locked 0x0005bd525ba0 (a java.lang.Object) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) - locked 0x0005bd5e81c0 (a com.mysql.jdbc.ServerPreparedStatement) at com.mysql.jdbc.Connection.closeAllOpenStatements(Connection.java:2126) at com.mysql.jdbc.Connection.realClose(Connection.java:4422) at com.mysql.jdbc.Connection.close(Connection.java:2098) - locked 0x0005bd522568 (a com.mysql.jdbc.Connection) at org.apache.tomcat.jdbc.pool.PooledConnection.disconnect(PooledConnection.java:320) Please help us on this. Could it be a problem with tomcat-jdbc.jar? Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Thursday, January 26, 2012 9:41 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/25/12 11:53 PM, gnath wrote: As you have suggested, i started collecting the thread dumps Thread dumps will set you free. Well, not really. Instead, they will tell you where your webapp is breaking, which usually means more work for you. But at least the hard part is done: finding out what's breaking. when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. That will definitely do it. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same configuration properties). Why? After this change, it seemed for few hours but started seeing in the logs where the Connection Pool jar could not give any connections and seems to be all the connections are busy. So we went ahead and added a configuration property 'removeAbandoned=true' in our Datasource configuration. I would go back to DBCP unless you think you need to switch for some reason. I suspect you are leaking database connections and don't have a suitable timeout for removal of lost database connections (or maybe didn't have that set up in the first place). You really need
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Hi Chris, Thanks a lot for looking into this and giving answers for all my questions. Sorry, i could not get chance to reply in time. As you have suggested, i started collecting the thread dumps when it happened again and we saw some kind of DBCP Connection pool issues leading to 'Too Many open files' issue. So we decided to replace the commons DBCP with tomcat-jdbc.jar (with same configuration properties). After this change, it seemed for few hours but started seeing in the logs where the Connection Pool jar could not give any connections and seems to be all the connections are busy. So we went ahead and added a configuration property 'removeAbandoned=true' in our Datasource configuration. We are still watching the performance and the server behavior after these changes. Will keep you posted on how things will turn out or if i see any further issues. thank you once again, I really appreciate your help. Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Monday, January 23, 2012 7:51 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/22/12 6:18 PM, gnath wrote: We have 2 connectors (one for http and another for https) using the tomcatThreadPool. I have the connectionTimeout=2 for http connector. However i was told that our https connector might not be used by the app as our loadbalancer is handling all the https traffic and just sending them to http connector. You might want to disable that HTTPS connector, but it's probably not hurting you at all in this case -- just a bit of wasted resources. If you are sharing a thread pool then there is no negative impact on the number of threads and/or open files that you have to deal with, here. the ulimit settings were increased from default 1024 to 4096 by our admin. not sure how he did that, but i see the count as 4096 when i do ulimit -a. Well, if your admin says it's right, I suppose it's right. for ulimit -n i see its 'unlimited'. That's good. for cat /proc/PID/limits, i get the following response: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 4096 4096 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 202752 202752 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 Those all look good to me. This morning Tomcat hung again but this time it dint say 'too many open files' in logs but i only see this below in catalina.out: org.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Jorg.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Hmm... When it hung(java process is still up), i ran few commands like lsof by PID and couple others. Next time, take a thread dump as well. The fact that Tomcat hung up without an OS problem (like Too Many Open Files) is probably not good. If this happens again with an apparent hang with no stack traces in the logs, take a thread dump and post it back here under a different subject. here is what i got: lsof -p PID| wc -l 1342 lsof | wc -l 4520 lsof -u USER| wc -l 1953 Hmm I wonder if you are hitting a *user* or even *system* limit of some kind (though a *NIX system with a hard limit of ~4500 file descriptors seems entirely unreasonable). I also wonder how many /processes/ and/or /threads/ you have running at once. After i kill java process the lsof for pid returned obviously to zero Of course. Is there any chance that the tomcat is ignoring the ulimit? Those limits are not self-imposed: the OS imposes those limits. Tomcat doesn't even know it's own ulimit (of any kind), so it will simply consume whatever resources you have configured it to use, and if it hits a limit, the JVM will experience some kind of OS-related error. , some people on web were saying something about setting this in catalina.sh. Setting what? ulimit? I'd do it in setenv.sh because that's a more
RE: Tomcat 6.0.35-SocketException: Too many open files issue with
Sorry to possibly state the obvious, but are there perhaps files that are not being closed? This can often happen if code goes into a catch, and a file is not closed. Best to have a finally block which checks if a file is open, and closes it. Robert Purvis -Original Message- From: gnath [mailto:gautam_exquis...@yahoo.com] Sent: 22 January 2012 08:01 To: users@tomcat.apache.org Subject: Tomcat 6.0.35-SocketException: Too many open files issue with Hello, We have been seeing SocketException: Too many open files in production environment(Linux OS running Tomcat 6.0.35 with sun's JDK 1.6.30) every day and requires a restart of Tomcat. When this happened for the first time, we searched online and found people suggesting to increase the file descriptors size and we increased to 4096. But still the problem persists. We have the Orion App Server also running on the same machine but usually during the day when we check the open file descriptor by command: ls -l /proc/PID/fd, its always less than 1000 combined for both Orion and Tomcat. Here is the exception we see pouring in the logs once it starts: This requires us to kill java process and restart tomcat. Our Tomcat configuration maxThreadCount is 500 with minSpareThreads=50 in server.xml SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:662) ulimit -a gives for the user where Tomcat is running. open files (-n) 4096 Please let me know what could be the issue here and how can i resolve this 'Too many open files' issue. Thanks -G This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit www.connectingforhealth.nhs.uk/nhsmail - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Hello Robert, We are talking about an app which has been grown since years and now migrated to Tomcat. Most of the main code is migrated to use spring jdbc and spring 3.0 mvc. Would you recommend some way that i can spot the issue from inspecting the open file descriptors once the server hangs/crashes so that i can check the code to see what you are saying? Please do let me know. All, please help if you can give me any hints from the information i provided down below. Thanks -G From: Purvis Robert (NHS CONNECTING FOR HEALTH) robert.pur...@nhs.net To: Tomcat Users List users@tomcat.apache.org; gnath gautam_exquis...@yahoo.com Sent: Monday, January 23, 2012 1:32 AM Subject: RE: Tomcat 6.0.35-SocketException: Too many open files issue with Sorry to possibly state the obvious, but are there perhaps files that are not being closed? This can often happen if code goes into a catch, and a file is not closed. Best to have a finally block which checks if a file is open, and closes it. Robert Purvis -Original Message- From: gnath [mailto:gautam_exquis...@yahoo.com] Sent: 22 January 2012 08:01 To: users@tomcat.apache.org Subject: Tomcat 6.0.35-SocketException: Too many open files issue with Hello, We have been seeing SocketException: Too many open files in production environment(Linux OS running Tomcat 6.0.35 with sun's JDK 1.6.30) every day and requires a restart of Tomcat. When this happened for the first time, we searched online and found people suggesting to increase the file descriptors size and we increased to 4096. But still the problem persists. We have the Orion App Server also running on the same machine but usually during the day when we check the open file descriptor by command: ls -l /proc/PID/fd, its always less than 1000 combined for both Orion and Tomcat. Here is the exception we see pouring in the logs once it starts: This requires us to kill java process and restart tomcat. Our Tomcat configuration maxThreadCount is 500 with minSpareThreads=50 in server.xml SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:662) ulimit -a gives for the user where Tomcat is running. open files (-n) 4096 Please let me know what could be the issue here and how can i resolve this 'Too many open files' issue. Thanks -G This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit www.connectingforhealth.nhs.uk/nhsmail
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
On Mon, 2012-01-23 at 05:33 -0800, gnath wrote: Hello Robert, We are talking about an app which has been grown since years and now migrated to Tomcat. Most of the main code is migrated to use spring jdbc and spring 3.0 mvc. Would you recommend some way that i can spot the issue from inspecting the open file descriptors once the server hangs/crashes so that i can check the code to see what you are saying? Please do let me know. Try running FindBugs on your code. In most cases, it will detect when you do not correctly close a file. Dan All, please help if you can give me any hints from the information i provided down below. Thanks -G From: Purvis Robert (NHS CONNECTING FOR HEALTH) robert.pur...@nhs.net To: Tomcat Users List users@tomcat.apache.org; gnath gautam_exquis...@yahoo.com Sent: Monday, January 23, 2012 1:32 AM Subject: RE: Tomcat 6.0.35-SocketException: Too many open files issue with Sorry to possibly state the obvious, but are there perhaps files that are not being closed? This can often happen if code goes into a catch, and a file is not closed. Best to have a finally block which checks if a file is open, and closes it. Robert Purvis -Original Message- From: gnath [mailto:gautam_exquis...@yahoo.com] Sent: 22 January 2012 08:01 To: users@tomcat.apache.org Subject: Tomcat 6.0.35-SocketException: Too many open files issue with Hello, We have been seeing SocketException: Too many open files in production environment(Linux OS running Tomcat 6.0.35 with sun's JDK 1.6.30) every day and requires a restart of Tomcat. When this happened for the first time, we searched online and found people suggesting to increase the file descriptors size and we increased to 4096. But still the problem persists. We have the Orion App Server also running on the same machine but usually during the day when we check the open file descriptor by command: ls -l /proc/PID/fd, its always less than 1000 combined for both Orion and Tomcat. Here is the exception we see pouring in the logs once it starts: This requires us to kill java process and restart tomcat. Our Tomcat configuration maxThreadCount is 500 with minSpareThreads=50 in server.xml SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:662) ulimit -a gives for the user where Tomcat is running. open files (-n) 4096 Please let me know what could be the issue here and how can i resolve this 'Too many open files' issue. Thanks -G This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful. Thank you for your co-operation. NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and GSi recipients NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit www.connectingforhealth.nhs.uk/nhsmail
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/22/12 6:18 PM, gnath wrote: We have 2 connectors (one for http and another for https) using the tomcatThreadPool. I have the connectionTimeout=2 for http connector. However i was told that our https connector might not be used by the app as our loadbalancer is handling all the https traffic and just sending them to http connector. You might want to disable that HTTPS connector, but it's probably not hurting you at all in this case -- just a bit of wasted resources. If you are sharing a thread pool then there is no negative impact on the number of threads and/or open files that you have to deal with, here. the ulimit settings were increased from default 1024 to 4096 by our admin. not sure how he did that, but i see the count as 4096 when i do ulimit -a. Well, if your admin says it's right, I suppose it's right. for ulimit -n i see its 'unlimited'. That's good. for cat /proc/PID/limits, i get the following response: Limit Soft Limit Hard Limit Units Max cpu time unlimitedunlimited seconds Max file size unlimitedunlimited bytes Max data size unlimitedunlimited bytes Max stack size10485760 unlimited bytes Max core file size0unlimited bytes Max resident set unlimitedunlimited bytes Max processes unlimitedunlimited processes Max open files4096 4096 files Max locked memory 3276832768 bytes Max address space unlimitedunlimited bytes Max file locksunlimitedunlimited locks Max pending signals 202752 202752 signals Max msgqueue size 819200 819200 bytes Max nice priority 00 Max realtime priority 00 Those all look good to me. This morning Tomcat hung again but this time it dint say 'too many open files' in logs but i only see this below in catalina.out: org.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Jorg.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Hmm... When it hung(java process is still up), i ran few commands like lsof by PID and couple others. Next time, take a thread dump as well. The fact that Tomcat hung up without an OS problem (like Too Many Open Files) is probably not good. If this happens again with an apparent hang with no stack traces in the logs, take a thread dump and post it back here under a different subject. here is what i got: lsof -p PID| wc -l 1342 lsof | wc -l 4520 lsof -u USER| wc -l 1953 Hmm I wonder if you are hitting a *user* or even *system* limit of some kind (though a *NIX system with a hard limit of ~4500 file descriptors seems entirely unreasonable). I also wonder how many /processes/ and/or /threads/ you have running at once. After i kill java process the lsof for pid returned obviously to zero Of course. Is there any chance that the tomcat is ignoring the ulimit? Those limits are not self-imposed: the OS imposes those limits. Tomcat doesn't even know it's own ulimit (of any kind), so it will simply consume whatever resources you have configured it to use, and if it hits a limit, the JVM will experience some kind of OS-related error. , some people on web were saying something about setting this in catalina.sh. Setting what? ulimit? I'd do it in setenv.sh because that's a more appropriate place for that kind of thing. I'm also interested in what the Internet has to say about what setting(s) to use. Please help with my ongoing issue.. its getting very hard to monitor the logs every minute and restarting whenever it hangs with these kind of issues. I very much appreciate your help in this. Did this just start happening recently? Perhaps with an upgrade of some component? If you think this might actually be related to the number of file handles being used by your thread pool, you might want to reduce the maximum number of threads for that thread pool: a slightly less responsive site is better than one that goes down all the time because of hard resource limits. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8dghAACgkQ9CaO5/Lv0PCmKQCfUaYfeoSkTRDKBppR4ZGFTWgI 8dEAoKgwy1BcKO6bC8nbbLWd6hn0a38N =TZCu -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/22/12 3:01 AM, gnath wrote: We have been seeing SocketException: Too many open files in production environment(Linux OS running Tomcat 6.0.35 with sun's JDK 1.6.30) every day and requires a restart of Tomcat. When this happened for the first time, we searched online and found people suggesting to increase the file descriptors size and we increased to 4096. But still the problem persists. We have the Orion App Server also running on the same machine but usually during the day when we check the open file descriptor by command: ls -l /proc/PID/fd, its always less than 1000 combined for both Orion and Tomcat. Here is the exception we see pouring in the logs once it starts: This requires us to kill java process and restart tomcat. Our Tomcat configuration maxThreadCount is 500 with minSpareThreads=50 in server.xml How many connectors do you have? If you have more than one connector with 500 threads, then you can have more threads than maybe you are expecting. SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:662) ulimit -a gives for the user where Tomcat is running. open files (-n) 4096 How did you set the ulimit for this user? Did you do it in a login script or something, or just at the command-line at some point? How about (-u) max user processes or threads-per-process or anything like that? Sometimes the Too many files open is not entirely accurate. What does 'cat /proc/PID/limits' show you? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk8cYZMACgkQ9CaO5/Lv0PC7+ACeMW3/jwhOUKB9RZ3u+dfN85jD NnMAoLU7QJ6DXKaI9Q/mPeEO6x9gXzx6 =Nd1d -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat 6.0.35-SocketException: Too many open files issue with
Thanks chris for looking into this. Here are answers for the questions you asked. We have 2 connectors (one for http and another for https) using the tomcatThreadPool. I have the connectionTimeout=2 for http connector. However i was told that our https connector might not be used by the app as our loadbalancer is handling all the https traffic and just sending them to http connector. the ulimit settings were increased from default 1024 to 4096 by our admin. not sure how he did that, but i see the count as 4096 when i do ulimit -a. for ulimit -n i see its 'unlimited'. for cat /proc/PID/limits, i get the following response: Limit Soft Limit Hard Limit Units Max cpu time unlimited unlimited seconds Max file size unlimited unlimited bytes Max data size unlimited unlimited bytes Max stack size 10485760 unlimited bytes Max core file size 0 unlimited bytes Max resident set unlimited unlimited bytes Max processes unlimited unlimited processes Max open files 4096 4096 files Max locked memory 32768 32768 bytes Max address space unlimited unlimited bytes Max file locks unlimited unlimited locks Max pending signals 202752 202752 signals Max msgqueue size 819200 819200 bytes Max nice priority 0 0 Max realtime priority 0 0 This morning Tomcat hung again but this time it dint say 'too many open files' in logs but i only see this below in catalina.out: org.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored Jorg.apache.tomcat.util.http.Parameters processParameters INFO: Invalid chunk starting at byte [0] and ending at byte [0] with a value of [null] ignored When it hung(java process is still up), i ran few commands like lsof by PID and couple others. here is what i got: lsof -p PID| wc -l 1342 lsof | wc -l 4520 lsof -u USER| wc -l 1953 After i kill java process the lsof for pid returned obviously to zero Is there any chance that the tomcat is ignoring the ulimit?, some people on web were saying something about setting this in catalina.sh. Please help with my ongoing issue.. its getting very hard to monitor the logs every minute and restarting whenever it hangs with these kind of issues. I very much appreciate your help in this. Thanks -G From: Christopher Schultz ch...@christopherschultz.net To: Tomcat Users List users@tomcat.apache.org Sent: Sunday, January 22, 2012 11:20 AM Subject: Re: Tomcat 6.0.35-SocketException: Too many open files issue with -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 G, On 1/22/12 3:01 AM, gnath wrote: We have been seeing SocketException: Too many open files in production environment(Linux OS running Tomcat 6.0.35 with sun's JDK 1.6.30) every day and requires a restart of Tomcat. When this happened for the first time, we searched online and found people suggesting to increase the file descriptors size and we increased to 4096. But still the problem persists. We have the Orion App Server also running on the same machine but usually during the day when we check the open file descriptor by command: ls -l /proc/PID/fd, its always less than 1000 combined for both Orion and Tomcat. Here is the exception we see pouring in the logs once it starts: This requires us to kill java process and restart tomcat. Our Tomcat configuration maxThreadCount is 500 with minSpareThreads=50 in server.xml How many connectors do you have? If you have more than one connector with 500 threads, then you can have more threads than maybe you are expecting. SEVERE: Socket accept failed java.net.SocketException: Too many open files at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408) at java.net.ServerSocket.implAccept(ServerSocket.java:462) at java.net.ServerSocket.accept(ServerSocket.java:430) at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:61) at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:352) at java.lang.Thread.run(Thread.java:662) ulimit -a gives for the user where Tomcat is running. open files (-n) 4096 How did you set the ulimit for this user? Did you do it in a login script or something, or just at the command-line at some point
