AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi all, I just want to give you an update about my problem THANKS TO ALL Felix gave me the final hint... I really had some connection not given back to the pool, so the connections kept open.. - :-( I thought I checked that. Since I fixed that and use removeAbandoned there was no more crash!!! Finally.. :-) I REALY appreciate your help. You guys are doing a really GREAT job! Thanks again.. David Kumar Hi Felix, 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. thanks for that hint, I will have a look at all of my databasepools.. Hopefully you're right and that is our problem.. thanks avid - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi Felix, 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. thanks for that hint, I will have a look at all of my databasepools.. Hopefully you're right and that is our problem.. thanks avid - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 3/18/13 2:07 AM, David Kumar wrote: 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. thanks for that hint, I will have a look at all of my databasepools.. Hopefully you're right and that is our problem.. Recommended reading: http://blog.christopherschultz.net/index.php/2009/03/16/properly-handling-pooled-jdbc-connections/ - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlFHMhEACgkQ9CaO5/Lv0PAgVwCfUoMRvSSqqrti566y/YipPTm8 b2QAoLIa3huGEVhwXEj+BnVdWAcrxUL7 =NhLI -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, It got stripped by the list. Can you post it somewhere and make the URL available? here you go: - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, It got stripped by the list. Can you post it somewhere and make the URL available? here you go: http://www.telestar.de/tmp10356/threaddump.txt thanks David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi David, Am 17.03.2013 16:17, schrieb David Kumar: Hey Rainer, It got stripped by the list. Can you post it somewhere and make the URL available? here you go: http://www.telestar.de/tmp10356/threaddump.txt almost all of your threads are waiting to get a database connection out of the pool. There are three possible scenarios, where this might happen. 1. You are abandoning db connections and thus are draining the pool (most likely in my eyes) 2. The pool is too small (unlikely, since no work seems to be done other than waiting for new connections) 3. Lock contention while going through borrowObject (unlikely, since no thread is doing a validation) So I would configure 'logAbandoned=true' and even 'removeAbandoned=true' and 'removeAbandonedTimeout=60' as stated on http://tomcat.apache.org/tomcat-7.0-doc/jndi-datasource-examples-howto.html. When your pool is getting drained, you will see a message in your log files, where you have taken connections out of the pool, but haven used them for a while. Than you can start to look for places, where those connections might have leaked. Regards Felix thanks David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Interesting. If the problem was too many sockets in CLOSE_WAIT, consider looking at the connectionLinger setting on your AJP connector's in Tomcat. Mark Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
sry I was wrong default is -1 what menas it is disabled, so I'm fine..?!? Interesting. If the problem was too many sockets in CLOSE_WAIT, consider looking at the connectionLinger setting on your AJP connector's in Tomcat. Mark Mhh.. I haven't heard about the options before.. But it sound like a great idea.. Default is -1. So what would be a good time, one minute? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 12.03.2013 11:58, David Kumar wrote: We got the connection_pool_timeout from here: https://community.jboss.org/wiki/OptimalModjk12Configuration I will have a look on the other recommended options Pick a source download of mod_jk. It contains an example configuration that should work pretty well. Don't choose the minimal example config, just the normal one. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 14.03.2013 10:04, David Kumar wrote: Hey, thanks for note.. Attached you can find a new list. So, java is keeping these connections in close_wait. close_wait for an AJP connection seen from Tomcat means the other side - mod_jk - has closed the connection, but not Tomcat. This is often due to a shorter Timeout on the mod_jk side than on the Tomcat side. It is not a problem per se, but it is if it happens for too many connections for a too long time. I vaguely remember that you have a 10 second socket_timeout in your workers.properties. That's typically bad. Look at the example config in the source mod_jk download. It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Rainer Jung wrote: On 14.03.2013 10:04, David Kumar wrote: Hey, thanks for note.. Attached you can find a new list. So, java is keeping these connections in close_wait. close_wait for an AJP connection seen from Tomcat means the other side - mod_jk - has closed the connection, but not Tomcat. This is often due to a shorter Timeout on the mod_jk side than on the Tomcat side. It is not a problem per se, but it is if it happens for too many connections for a too long time. I vaguely remember that you have a 10 second socket_timeout in your workers.properties. That's typically bad. Look at the example config in the source mod_jk download. It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. Hi Rainer, a question to you : In a previous post, David posted the output of netstat -t -pan. In that output, there are about 1900 connections from Apache to Tomcat's AJP connectors, in state TIME_WAIT. As far as I know, this indicates that the connection is closed from the point of view of Apache, and this TIME_WAIT should last only a few sec. maximum, and then should go away. Why does he have so many though ? I can't see anything like that on any of my servers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
close_wait for an AJP connection seen from Tomcat means the other side - mod_jk - has closed the connection, but not Tomcat. This is often due to a shorter Timeout on the mod_jk side than on the Tomcat side. It is not a problem per se, but it is if it happens for too many connections for a too long time. I vaguely remember that you have a 10 second socket_timeout in your workers.properties. That's typically bad. Look at the example config in the source mod_jk download. I will do that. It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. I try to, currently we have a watchdog running, checking every 2 minutes if the tomcats are available at 7009/9009 and http. If one of the tomcats are not responding it will be restarted Thanks David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 15.03.2013 11:47, André Warnier wrote: Rainer Jung wrote: On 14.03.2013 10:04, David Kumar wrote: Hey, thanks for note.. Attached you can find a new list. So, java is keeping these connections in close_wait. close_wait for an AJP connection seen from Tomcat means the other side - mod_jk - has closed the connection, but not Tomcat. This is often due to a shorter Timeout on the mod_jk side than on the Tomcat side. It is not a problem per se, but it is if it happens for too many connections for a too long time. I vaguely remember that you have a 10 second socket_timeout in your workers.properties. That's typically bad. Look at the example config in the source mod_jk download. It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. Hi Rainer, a question to you : In a previous post, David posted the output of netstat -t -pan. In that output, there are about 1900 connections from Apache to Tomcat's AJP connectors, in state TIME_WAIT. As far as I know, this indicates that the connection is closed from the point of view of Apache, and this TIME_WAIT should last only a few sec. maximum, and then should go away. Why does he have so many though ? I can't see anything like that on any of my servers. TIME_WAIT is a state which means the connection is closed by both sides but the TCP stack still blocks the (server port, server ip, client port, client IP) quadruple for reuse to prevent long overdue packets of the old connection to disturb the new connection. It doesn't prevent new connections from being created nor are there still process resources on client or server being used for those connections (threds, buffers). It is just an entry in the OS TCP connection hash. The TIME_WAIT state is only used on the side of the connection that started the connection shutdown, i.e. that first send the FIN. The other side doesn't go into TIME_WAIT. If your clients typically start the connection shutdown you won't see the TIME_WAIT. Connections are kept in TIME_WAIT for seconds to about a minute depending on your OS and TCP tuning. For a service which uses a lot of new connections per second and initiates the shutdown for them itself it is not uncommon to see several hundreds to thousands of TIME_WAIT connections. If the count goes up into the range of 10.000-50.000 you could start running into inefficiencies of the TCP connection hash handling (depending on the OS). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 15.03.2013 11:57, David Kumar wrote: It could be, that your requests in Tomcat got stuck and Tomcat still is in the state of working on the requests, therefore keeping the connection open to send back stuff finally, whereas mod_jk has already timed out. To check for that, take a couöple of threa dumps (not: heap dumps) of the running Tomcat process while the close_wait problem is visible. Check what your Tomcat threads are currently doing, e.g. are they mostly sitting idle in the thread pool or executor, or are many of them deep in your application stacks and waiting for database, locks or other stuff. I try to, currently we have a watchdog running, checking every 2 minutes if the tomcats are available at 7009/9009 and http. If one of the tomcats are not responding it will be restarted Please please before restarting take two thread dumps. On Linux/Unix a kill -QUIT will do that (and write the result to STDOUT, which depending on your setup likely goes to catalina.out). The QUIT signal does *not* terminate the JVM, it just quickly writes out the stacks of all threads. The interruption for the running JVM is minimal, so you can already try how this works before the process gets into the critical state. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, Please please before restarting take two thread dumps. On Linux/Unix a kill -QUIT will do that (and write the result to STDOUT, which depending on your setup likely goes to catalina.out). The QUIT signal does *not* terminate the JVM, it just quickly writes out the stacks of all threads. The interruption for the running JVM is minimal, so you can already try how this works before the process gets into the critical state. great I just tried a kill -QUIT in our watchdog -- pretty fast. Next time we got our problem I will send the threaddump to you guys. thanks a lot David - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Rainer, attached you can find a Threaddump. Just rename it to .zip. I'm not sure waht all the stuff at the dump means. but I'm sure you know.. :-) thanks Please please before restarting take two thread dumps. On Linux/Unix a kill -QUIT will do that (and write the result to STDOUT, which depending on your setup likely goes to catalina.out). The QUIT signal does *not* terminate the JVM, it just quickly writes out the stacks of all threads. The interruption for the running JVM is minimal, so you can already try how this works before the process gets into the critical state. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 15.03.2013 15:44, David Kumar wrote: Hey Rainer, attached you can find a Threaddump. Just rename it to .zip. I'm not sure waht all the stuff at the dump means. but I'm sure you know.. :-) It got stripped by the list. Can you post it somewhere and make the URL available? Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 DAvid, On 3/13/13 3:01 PM, David Kumar wrote: Hey, right no we're having our Problem. I tried gc through jconsole -- no changes and still a lot CLOSE_WAIT. So it is not a GC Problem and disablereuse doesn't work either.. Any other ideas? What do you guy think about proxy_ajp instead of jk? What are the advantages of proxy_ajp? mod_proxy_ajp uses the identical protocol to mod_jk: they both use AJP, and the code on the Tomcat side is identical. So, if you think this is a Java problem, switching from mod_jk - mod_proxy_ajp is unlikely to change anything at all. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlFDemcACgkQ9CaO5/Lv0PAt0ACZAZg5UqqAeyEhpuOcssa02p4d DjYAn0IwtT+RJ6pdw+jMX0cy6A8gDhai =A38V -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hi. Note: try to avoid top-posting, on this list. (top-posting means responding on top of the previous message, like I am doing just now). The reason why we ask not to do that, is that it forces whoever is reading your message, to scroll down and try to figure out to what question(s) you are answering. Answer in the text, just below the question, or at the end (and remove what is no longer relevant). This way, someone else can easily follow the logic flow of the matter. End of Note. David Kumar wrote: Hey, I'm using: lsof -u tomcat Both of our tomcats are runing wiht user tomcat.. Attached you can find a sample from today 07:00 am (GMT+1) How about a netstat -t -pan ? it's a bit easier to find the relevant connections. If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd (if it is not evident from the list). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: Hey, I'm using: lsof -u tomcat example of netstat output on one of our own systems right now : 1) first one vovm1:~# netstat -t -pan | grep -v ESTAB Active Internet connections (servers and established) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp0 0 0.0.0.0:32032 0.0.0.0:* LISTEN 2999/usbsrvd tcp0 0 0.0.0.0:20224 0.0.0.0:* LISTEN 2965/inetd tcp0 0 0.0.0.0:11022 0.0.0.0:* LISTEN 9179/perl tcp0 0 127.0.0.1:111 0.0.0.0:* LISTEN 2172/portmap tcp0 0 0.0.0.0:54321 0.0.0.0:* LISTEN 18279/python tcp0 0 0.0.0.0:49490.0.0.0:* LISTEN 23225/munin-node tcp0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2431/sshd tcp0 0 0.0.0.0:23 0.0.0.0:* LISTEN 2965/inetd tcp0 0 0.0.0.0:43961 0.0.0.0:* LISTEN - tcp0 0 192.168.20.43:250.0.0.0:* LISTEN 2946/exim4 tcp0 0 127.0.0.1:250.0.0.0:* LISTEN 2946/exim4 tcp0 0 0.0.0.0:11002 0.0.0.0:* LISTEN 2965/inetd tcp0 0 0.0.0.0:46911 0.0.0.0:* LISTEN 2183/rpc.statd tcp0 0 127.0.0.1:11002 127.0.0.1:40739 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:40735 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:40728 TIME_WAIT - tcp0 0 127.0.0.1:11002 127.0.0.1:40724 TIME_WAIT - tcp1 0 127.0.0.1:54321 127.0.0.1:37916 CLOSE_WAIT 18279/python tcp0 0 127.0.0.1:43797 127.0.0.1:8009 TIME_WAIT - tcp0 0 192.168.20.43:56603 192.168.20.80:111 TIME_WAIT - tcp6 0 0 :::8009 :::*LISTEN 9206/jsvc tcp6 0 0 :::80 :::*LISTEN 10730/apache2 tcp6 0 0 :::8180 :::*LISTEN 9206/jsvc tcp6 0 0 :::22 :::*LISTEN 2431/sshd tcp6 0 0 :::11100:::*LISTEN 9133/java tcp6 0 0 :::11101:::*LISTEN 9160/java tcp6 0 0 127.0.0.1:11100 127.0.0.1:38143 TIME_WAIT - tcp6 0 0 127.0.0.1:40725 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:11101 127.0.0.1:54664 TIME_WAIT - tcp6 1 0 127.0.0.1:8009 127.0.0.1:35156 CLOSE_WAIT 9206/jsvc tcp6 0 0 212.85.38.148:8062.99.208.50:49516 TIME_WAIT - tcp6 0 0 212.85.38.148:8062.155.235.89:50654 TIME_WAIT - tcp6 0 0 127.0.0.1:11101 127.0.0.1:54649 TIME_WAIT - tcp6 1 0 127.0.0.1:8009 127.0.0.1:33159 CLOSE_WAIT 9206/jsvc tcp6 1 0 127.0.0.1:8009 127.0.0.1:60682 CLOSE_WAIT 9206/jsvc tcp6 1 0 127.0.0.1:8009 127.0.0.1:33138 CLOSE_WAIT 9206/jsvc tcp6 0 0 127.0.0.1:40743 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:40733 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:40740 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:11100 127.0.0.1:38136 TIME_WAIT - tcp6 0 0 127.0.0.1:11101 127.0.0.1:54656 TIME_WAIT - tcp6 0 0 127.0.0.1:11101 127.0.0.1:54652 TIME_WAIT - tcp6 0 0 127.0.0.1:11100 127.0.0.1:50778 TIME_WAIT - tcp6 0 0 127.0.0.1:40732 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:40742 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:11101 127.0.0.1:54667 TIME_WAIT - tcp6 0 0 127.0.0.1:40729 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:40736 127.0.0.1:11002 TIME_WAIT - tcp6 0 0 127.0.0.1:11100 127.0.0.1:38140 TIME_WAIT - vovm1:~# (The grep -v ESTAB is to eliminate the ESTABLISHED connections, which are maybe irrelevant here). The jsvc process here is the wrapper which wraps the JVM and Tomcat to allow them to use ports below 1024 and still run as non-root. For practical purposes, consider it as tomcat. As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state also.
AW: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey André, André Warnier wrote: The jsvc process here is the wrapper which wraps the JVM and Tomcat to allow them to use orts below 1024 and still run as non-root. For practical purposes, consider it as tomcat. As you can see, there are some AJP connections (local port 8009) in the CLOSE_WAIT state also. It is a normal state of a TCP connection. What is less normal, is that they would remain in that state for a long time. That means that one of the sides of the TCP connection is not doing what it should do. In my experience, under Linux, this can become a problem if you have hundreds of these. At some point, the TCP stack becomes unresponsive and does not allow any new connection. Sometimes, it can happen because the client side in the connection (the one who initially creates the connection to a server's listening socket), discards a Java object which contains an open low-level OS socket object with a connection still open, without explicitly closing this connection first. The discarded (and unreachable) Java object is then still sitting in the heap for a while, and only a GC will actually destroy it and as a side-effect close the underlying connection. That is why I was asking you to force a GC, to see if this made your CLOSE_WAIT connections disappear. Apparently in your case it doesn't, which mean that yours is another case. Now, instead of restarting Tomcat or doing a GC, have you tried to restart Apache ? or maybe just do a /etc/init.d/apache2 reload. That may kill off and restart the Apache children processes, and clean up their mod_jk connections to Tomcat. I tried to just restart the Apache, didn't work. As suggested I tried to use the option -DisableReuse. That did make our situation a lot more horrible instead of days / week we it took just minutes to shut all AJP ports down. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 14/03/2013 13:10, David Kumar wrote: I tried to just restart the Apache, didn't work. As suggested I tried to use the option -DisableReuse. That did make our situation a lot more horrible instead of days / week we it took just minutes to shut all AJP ports down. Interesting. If the problem was too many sockets in CLOSE_WAIT, consider looking at the connectionLinger setting on your AJP connector's in Tomcat. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Dienstag, 12. März 2013 09:54
An: Tomcat Users List
Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we are still having that issue.
But we could manage to figure out some more stuff.
We made a Tomcat and Java update since that time we had our problem a few
times again, also we did some reconfiguration with connectors etc.
The last 2 times we where able to see, that both tomcat by them self where
alive. Just ajp on both where dead. We couldn't make a connection either
trough 7009 nor 9009. An with our openFiles trick we found a lot of
close_wait again, e.g. 200 for 9009. I left the second tomcat on this state
for a few ours just to see, what happens. The count of 200 connection with
close_wait was kept until a reboot of the tomcat.
Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage
Collection.
There are a number of tools that allow to do that.
One command-line one which I found practical is jmxsh, here :
http://code.google.com/p/jmxsh/
If when you do a Major GC, these CLOSE_WAIT connections disappear, you will
have learned
something about their origin.
And if then - without restarting Tomcat - you can connect again via the AJP
ports, you'll
have learned something else.
Go do it and report.
I would say with some of our reconfiguration we managed to stop increasing
connections. But still we are not sure why our ajp connections dying..
Here is our connector out of Server.xml:
Connector port=9009 protocol=AJP/1.3 redirectPort=9443
maxThreads=200 connectionTimeout=60 /
worker.properties:
worker.tomcatX.host=localhost
worker.tomcatX.type=ajp13
worker.tomcatX.fail_on_status=404
worker.tomcatX.lbfactor=1
worker.tomcatX.ping_timeout=1000
worker.tomcatX.ping_mode=A
worker.tomcatX.socket_timeout=10
worker.tomcatX.connection_pool_timeout=600
worker.tomcat1.reference=worker.tomcatX
worker.tomcat1.port=7009
worker.tomcat2.reference=worker.tomcatX
worker.tomcat2.port=9009
worker.loadbalancer.type=lb
worker.loadbalancer.balance_workers=tomcat1,tomcat2
worker.status.type=status
Hopefully one of you guys can give us a hint to fix that problem.
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Dienstag, 22. Januar 2013 07:36
An: Tomcat Users List
Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open
Hey,
last friday I changed our configuration to use a executor.
Here is what I did:
Connector port=7009 protocol=AJP/1.3 redirectPort=8443
maxThreads=200 executor=active-executor /
Executor name=active-executor namePrefix=activeThread- maxThreads=200
minSpareThreads=30 maxIdleTime=6 /
Connector port=7080 protocol=HTTP/1.1
connectionTimeout=2
redirectPort=8443 executor=active-executor /
The second tomcat has same configuration besides ports..
Until yesterday it worked like a charm. But at late afternoon one of the
tomcats failed again..
I couldn't start the garbagecollection so far..
Any other ideas?
Thanks
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Freitag, 18. Januar 2013 11:19
An: Tomcat Users List; Tomcat Users List
Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections
keept open
Hey,
I do that at next deployment. -- I Thursday...
So far I'm trying executor for tomcat. As far I read, when I'm using
connectors idle process are forced to be close..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey,
right no we're having our Problem.
I tried gc through jconsole -- no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of
proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
BTW... after a while CLOSE_WAIT changes to can't identify protocol
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
Tel.: + 49 (0) 6592 / 712 -2826
Tel.: + 49 (0) 2676 / 9520 -183
Fax: + 49 (0) 6592 / 712 -2829
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar
Gesendet: Mittwoch, 13. März 2013 20:01
An: 'Tomcat Users List'
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
right no we're having our Problem.
I tried gc through jconsole -- no changes and still a lot CLOSE_WAIT.
So it is not a GC Problem and disablereuse doesn't work either..
Any other ideas?
What do you guy think about proxy_ajp instead of jk? What are the advantages of
proxy_ajp?
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: David Kumar [mailto:dku...@telestar.de]
Gesendet: Mittwoch, 13. März 2013 14:32
An: Tomcat Users List
Betreff: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
Hey,
I'm really not sure if that tool does anything, because there is no feedback..
Funny: Just a few minutes ago a colleague showed ma a tool called something
like VM monitor on his mac. It look a kind of similar to jconsole.exe. Next
time I will try to make GC with this tool.
Meanwhile I reduced the connectionTimeout, reconfigured maxClients and added
-DisableReuse to jkpropertys.
Lets see what happens...
Thanks a lot..
Mit freundlichen Grüßen
David Kumar
Softwareentwickler, B. Sc.
Abteilung Infotech - Interaktiv
TELESTAR-DIGITAL GmbH
Am Weiher 14
D-56766 Ulmen
http://www.telestar.de/
-Ursprüngliche Nachricht-
Von: André Warnier [mailto:a...@ice-sa.com]
Gesendet: Mittwoch, 13. März 2013 12:20
An: Tomcat Users List
Betreff: Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections
keept open
David Kumar wrote:
Hey,
we just had that problem again. I did that GC trick, it didn't work out well.
I used:
set hosts [list]
lappend hosts {localhost:7008}
lappend hosts {localhost:9008}
# ...add as many as you want...
foreach {host} $hosts {
set parts [split $host :]
set hostname [lindex $parts 0]
set port [lindex $parts 1]
# for each host...
# Connect to it.
jmx_connect -h $hostname -p $port
# Invoke the garbage collector.
jmx_invoke -n -m java.lang:type=Memory gc
# Close this connection
jmx_close
}
I found that while googleing.
But it doesn't have any effect. Any other ideas?
Did it really connect with the Tomcat JVM and really do anything ?
Is your JVM started with options that allow JMX interaction ?
Also, there are ways to start the JVM with parameters telling it to log the
Garbage
Collections to a logfile. Make sure that you do that, and check the logfile to
see if it
really does a GC when you ask it to.
Note that there are also other ways to trogger a GC than using jmxsh. I was
only pointing
you to that tool as an easy-to-script, non-GUI tool.
But in your java directory (at least the JDK/bin), there should also be a
jconsole
program which allows you to do pretty much the same interactively with a GUI.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: BTW... after a while CLOSE_WAIT changes to can't identify protocol Huh ? Which command are you using to see the connections in CLOSE_WAIT state ? (and maybe paste a sample here) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Dienstag, 22. Januar 2013 07:36 An: Tomcat Users List Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote: Just read this email.. :-) I figured out we are not using executor connector... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 09:11 An: Tomcat Users List Betreff: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection. There are a number of tools that allow to do that. One command-line one which I found practical is jmxsh, here : http://code.google.com/p/jmxsh/ If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned something about their origin. And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll have learned something else. Go do it and report. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Dienstag, 22. Januar 2013 07:36 An: Tomcat Users List Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 10:07 An: Tomcat Users List Betreff: Re: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David, (and sorry for top-posting here) just to verify something. Can you trigger a Major Garbage Collection at the Tomcat JVM level, at a moment when you have all these connections in CLOSE_WAIT, and see if they disappear after the GC ? If yes, it may give a good clue about where all these CLOSE_WAITs are coming from. David Kumar wrote:
Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 12/03/2013 06:53, David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / Only 200 threads on the Tomcat side. If httpd's MaxClients/MaxRequestWorkers is greater than 200 you may get thread starvation in Tomcat. worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 That is a really bad idea. A single 404 and the entire Tomcat instance gets taken out of the loadbalancer for 60 seconds. Hello DOS attack. worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 10 minutes is a long time to keep a persistent connection around. With even a moderate load you'll easily get to 200 connections in a 10 minute period. worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Do one of the following: 1. Increase maxThreads in Tomcat's connector to MaxRequestWorkers 2. Use JkOptions +DisableReuse (ignore the performance warnings) 3. Reduce the connection_pool_timeout Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey Mark, thanks for reply. I do have some more questions as the main configuration i not done by myself. We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it? Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients? We got the connection_pool_timeout from here: https://community.jboss.org/wiki/OptimalModjk12Configuration I will have a look on the other recommended options Thanks.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Dienstag, 12. März 2013 10:25 An: Tomcat Users List Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open On 12/03/2013 06:53, David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / Only 200 threads on the Tomcat side. If httpd's MaxClients/MaxRequestWorkers is greater than 200 you may get thread starvation in Tomcat. worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 That is a really bad idea. A single 404 and the entire Tomcat instance gets taken out of the loadbalancer for 60 seconds. Hello DOS attack. worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 10 minutes is a long time to keep a persistent connection around. With even a moderate load you'll easily get to 200 connections in a 10 minute period. worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Do one of the following: 1. Increase maxThreads in Tomcat's connector to MaxRequestWorkers 2. Use JkOptions +DisableReuse (ignore the performance warnings) 3. Reduce the connection_pool_timeout Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
On 12/03/2013 10:58, David Kumar wrote: Hey Mark, thanks for reply. I do have some more questions as the main configuration i not done by myself. We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it? Correct. Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients? Increase maxThreads to 400. Mark We got the connection_pool_timeout from here: https://community.jboss.org/wiki/OptimalModjk12Configuration I will have a look on the other recommended options Thanks.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: Mark Thomas [mailto:ma...@apache.org] Gesendet: Dienstag, 12. März 2013 10:25 An: Tomcat Users List Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open On 12/03/2013 06:53, David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / Only 200 threads on the Tomcat side. If httpd's MaxClients/MaxRequestWorkers is greater than 200 you may get thread starvation in Tomcat. worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 That is a really bad idea. A single 404 and the entire Tomcat instance gets taken out of the loadbalancer for 60 seconds. Hello DOS attack. worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 10 minutes is a long time to keep a persistent connection around. With even a moderate load you'll easily get to 200 connections in a 10 minute period. worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Do one of the following: 1. Increase maxThreads in Tomcat's connector to MaxRequestWorkers 2. Use JkOptions +DisableReuse (ignore the performance warnings) 3. Reduce the connection_pool_timeout Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey, thanks.. I downloaded jmxsh... I keep you guy up to date.. thanks... Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Dienstag, 12. März 2013 09:54 An: Tomcat Users List Betreff: Re: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open David Kumar wrote: Hey, we are still having that issue. But we could manage to figure out some more stuff. We made a Tomcat and Java update since that time we had our problem a few times again, also we did some reconfiguration with connectors etc. The last 2 times we where able to see, that both tomcat by them self where alive. Just ajp on both where dead. We couldn't make a connection either trough 7009 nor 9009. An with our openFiles trick we found a lot of close_wait again, e.g. 200 for 9009. I left the second tomcat on this state for a few ours just to see, what happens. The count of 200 connection with close_wait was kept until a reboot of the tomcat. Instead of rebooting Tomcat, try to force the Tomcat JVM to do a Major Garbage Collection. There are a number of tools that allow to do that. One command-line one which I found practical is jmxsh, here : http://code.google.com/p/jmxsh/ If when you do a Major GC, these CLOSE_WAIT connections disappear, you will have learned something about their origin. And if then - without restarting Tomcat - you can connect again via the AJP ports, you'll have learned something else. Go do it and report. I would say with some of our reconfiguration we managed to stop increasing connections. But still we are not sure why our ajp connections dying.. Here is our connector out of Server.xml: Connector port=9009 protocol=AJP/1.3 redirectPort=9443 maxThreads=200 connectionTimeout=60 / worker.properties: worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=600 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status Hopefully one of you guys can give us a hint to fix that problem. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Dienstag, 22. Januar 2013 07:36 An: Tomcat Users List Betreff: AW: AW: AW: ajp on 7009 and 9009 : connections keept open Hey, last friday I changed our configuration to use a executor. Here is what I did: Connector port=7009 protocol=AJP/1.3 redirectPort=8443 maxThreads=200 executor=active-executor / Executor name=active-executor namePrefix=activeThread- maxThreads=200 minSpareThreads=30 maxIdleTime=6 / Connector port=7080 protocol=HTTP/1.1 connectionTimeout=2 redirectPort=8443 executor=active-executor / The second tomcat has same configuration besides ports.. Until yesterday it worked like a charm. But at late afternoon one of the tomcats failed again.. I couldn't start the garbagecollection so far.. Any other ideas? Thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: David Kumar [mailto:dku...@telestar.de] Gesendet: Freitag, 18. Januar 2013 11:19 An: Tomcat Users List; Tomcat Users List Betreff: AW: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open Hey, I do that at next deployment. -- I Thursday... So far I'm trying executor for tomcat. As far I read, when I'm using connectors idle process are forced to be close.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 18. Januar 2013 11:10 An: Tomcat Users List Betreff: Re: AW: AW: ( ajp on 7009 and 9009 not afs3-rmtsys): connections keept open David Kumar wrote: Hey André, are you talking about running System.gc()? Yes. That should be possible.. Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14
Re: AW: AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 David, On 3/12/13 6:58 AM, David Kumar wrote: We are using Apache 2.2.16 on Debian. Therefore MaxRequestWorkers is called MaxClients, isn't it? That depends upon the MPM you are using. I have Debian-based systems with both prefork and worker MPMs configured (separately). Currently it is set to 312. So if we have two tomcats with 200 thread each MaxClients is to low? I either should reduce the Thread at the connector or increase MaxClients? If you have MaxClients=312, then your Tomcat configuration should be at least that big, even if you have two back-end Tomcats. What happens if one of your Tomcats dies (or gets a 404 LOL)? All your traffic will go to the other Tomcat: that means 312 connections to a single back-end node. So you either need to adjust your MaxClients/MaxRequestWorkers /down/ so that a single Tomcat can handle those connections, or you need to increase your maxThreads on Tomcat to handle the potential client load from httpd. If you have multiple httpd instances, then you need to multiple everything by N: MaxClients * N = maxThreads. I would set maxThreads to be a bit bigger, anyway, because then you can do nice things like test to see if a Tomcat is available outside of httpd. You don't want MaxClients == maxThreads and then not be able to ping your server due to thread starvation. Finally, if you have more than 8192 possible incoming connections from all your httpd instances, you may have to adjust your maxConnections setting (default is 8192 for APR and 1 for NIO... default is maxThreads for BIO which is the only thing that makes any sense at all). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlE/U3gACgkQ9CaO5/Lv0PD1XACdG0l/hyIBXh1CoE5Yp6XRf81P D+kAniaCJT0KSKfSl7SSs6WLETCAB1Zx =hqwF -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
