AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
hey, Yes. Whether this is the java that runs tomcat is not immediately evident. But it also seems that they concern port numbers like 7009, 9009 etc. which must be your AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat. The first observation hat I would make is that this looks like a pretty busy system. Then, there are 2100+ lines in that list. Let's assume first that they are all internal connections (which probable they are not, but for simplicity), so that there are actually 2 lines for the same connection, like these : tcp0 0 127.0.0.1:39872 127.0.0.1:7009 FIN_WAIT2 20766/apache2 . .. tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 CLOSE_WAIT 20564/java .. So let's say that we have in total about 1000 connections active in some state. Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%. That does not really sound pathological to me, considering that CLOSE_WAIT is a normal state through which any TCP connection goes at some point. If you look again at the above connection, and if I am not confusing my clients and servers again, it looks here like : - Apache http (PID 20766), as a client, has established a connection to port 7009 of the server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of Apache, to the AJP Connector in Tomcat. - A while later, Apache closes the connection. It sends a FIN packet to Tomcat, and waits for Tomcat to acknowledge this FIN packet. - when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes to the CLOSE_WAIT state. - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2. (and that is the current state of the above connection) - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return. - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it discards that connection. If there are connections that remain in the CLOSE_WAIT state for very long, it means that something in the last 2 steps above is not working. Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an AJP port of Tomcat. This seems to suggest that the Apache side is working as it should. But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on the Tomcat side, that connection is never properly close()'d. But this is the request/response connection, so I doubt that there would be a bug there, otherwise we'd have problems reports filling this list every day. Might there be that there is somewhere a discrepancy between the keep-alive settings, between Apache and Tomcat ? or maybe your server is just not strong enough for the load you are putting on it ? According to vmtools the server is boring most times.. I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless? Attached you can find our configuration files, maybe you can find a / some misconfigurations? I would agree that we have maybe a kind of misconfigured timeout problem... thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ # # Based upon the NCSA server configuration files originally by Rob McCool. # # This is the main Apache server configuration file. It contains the # configuration directives that give the server its instructions. # See http://httpd.apache.org/docs/2.2/ for detailed information about # the directives. # # Do NOT simply read the instructions in here without understanding # what they do. They're here only as hints or reminders. If you are unsure # consult the online docs. You have been warned. # # The configuration directives are grouped into three basic sections: # 1. Directives that control the operation of the Apache server process as a # whole (the 'global environment'). # 2. Directives that define the parameters of the 'main' or 'default' server, # which responds to requests that aren't handled by a virtual host. # These directives also provide default values for the settings # of all virtual hosts. #
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: hey, Yes. Whether this is the java that runs tomcat is not immediately evident. But it also seems that they concern port numbers like 7009, 9009 etc. which must be your AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat. The first observation hat I would make is that this looks like a pretty busy system. Then, there are 2100+ lines in that list. Let's assume first that they are all internal connections (which probable they are not, but for simplicity), so that there are actually 2 lines for the same connection, like these : tcp0 0 127.0.0.1:39872 127.0.0.1:7009 FIN_WAIT2 20766/apache2 . .. tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 CLOSE_WAIT 20564/java .. So let's say that we have in total about 1000 connections active in some state. Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%. That does not really sound pathological to me, considering that CLOSE_WAIT is a normal state through which any TCP connection goes at some point. If you look again at the above connection, and if I am not confusing my clients and servers again, it looks here like : - Apache http (PID 20766), as a client, has established a connection to port 7009 of the server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of Apache, to the AJP Connector in Tomcat. - A while later, Apache closes the connection. It sends a FIN packet to Tomcat, and waits for Tomcat to acknowledge this FIN packet. - when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes to the CLOSE_WAIT state. - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2. (and that is the current state of the above connection) - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return. - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it discards that connection. If there are connections that remain in the CLOSE_WAIT state for very long, it means that something in the last 2 steps above is not working. Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an AJP port of Tomcat. This seems to suggest that the Apache side is working as it should. But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on the Tomcat side, that connection is never properly close()'d. But this is the request/response connection, so I doubt that there would be a bug there, otherwise we'd have problems reports filling this list every day. Might there be that there is somewhere a discrepancy between the keep-alive settings, between Apache and Tomcat ? or maybe your server is just not strong enough for the load you are putting on it ? According to vmtools the server is boring most times.. What do you mean, boring ? (like German langweilig ?). If you have 2000+ connections there in one state or another, that should mean that the system is usually pretty busy, I would think. On my production machines (vSphere VM's, Linux, Apache+Tomcat, +database), some of them pretty busy, I never see more than 200-300 connections with netstat. That is all states together, with netstat -t -pan | wc -l). Granted, I am not running Yahoo or Facebook, but still, these are public Internet servers, fairly sollicited. I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless? Let's look at the various options in the Connector : First, the -1 in some options means unlimited or forever, it doesn't mean disabled. Second, some parameters say that if they are not specified, they default to the value of another parameter, so you should check really what this other parameter is set to. Attached you can find our configuration files, maybe you can find a / some misconfigurations? I'll have a look. I would agree that we have maybe a kind of misconfigured timeout problem... thanks Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/
AW: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Hey André, According to vmtools the server is boring most times.. What do you mean, boring ? (like German langweilig ?). -- Jep prety idle.. :-) If you have 2000+ connections there in one state or another, that should mean that the system is usually pretty busy, I would think. On my production machines (vSphere VM's, Linux, Apache+Tomcat, +database), some of them pretty busy, I never see more than 200-300 connections with netstat. That is all states together, with netstat -t -pan | wc -l). Granted, I am not running Yahoo or Facebook, but still, these are public Internet servers, fairly sollicited. I almost totally agree, but I'm not sure if it is a AJP fault. Because most time everything is alright, so all of theses steps are done. Just sometimes after a arbitrarily amount of time ajp / tomcat runs crazy. And in my sample there is the sourceport 9009 / java. It looks for me that apache didn't received the ACK from tomcat after CLOSE_WAIT or it just don't care because apache thought this connections is already closed? Let's assume that is the case.. Isn't there a option to tell tomcat to wait max. time for a response and after that time kill the connection? As far as I understand connectionTimeout is just for the initial request, so after Apache did answer in first correctly connectionTimeout is valueless? Let's look at the various options in the Connector : First, the -1 in some options means unlimited or forever, it doesn't mean disabled. Second, some parameters say that if they are not specified, they default to the value of another parameter, so you should check really what this other parameter is set to. According to Tomcat Configuration Reference 5.5 : connectionLinger The number of milliseconds during which the sockets used by this Connector will linger when they are closed. The default value is -1 (socket linger is disabled). According to Tomcat Configuration Reference 7.0.37 : connectionLinger The number of seconds during which the sockets used by this Connector will linger when they are closed. If not specified, the JVM default will be used. As I'm at 7.0.37 I have to figure out where an how to set linger options at the JVM. I'll have a look. thx - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: Attached you can find our configuration files, maybe you can find a / some misconfigurations? I'll have a look. Some notes : - according to the comments in your Apache config file, you are using the worker MPM. But are you sure ? what does /usr/sbin/apache2ctl -l say ? - we are missing the contents of your /etc/apache2/workers.properties file - we are missing the JkMounts or equivalent that you are doing from Apache to Tomcat. related questions : you have 2 tomcats. Are you doing load-balancing ? or are you just sending some URLs to tomcat1 and other to tomcat2 ? Are you proxying everything to the tomcats, or is the Apache httpd front-end serving some URLs on its own ? if so, what proportion ? - how many hits per second/minute/hour (any of them, approx.) is your server handling ? - you do not have any specific timeout parameters set in your Tomcat AJP Connectors, which is basically a good thing : better to leave the defaults in place, than to start playing with settings that you do not really understand, and make things worse. But you do set one : connectionTimeout=20. Which sounds extremely high to me. Re: http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html It means that when a client connects to that Connector, a Tomcat thread will be started to handle this connection; then the thread will wait on the connection, to read the request. If the request does not appear, this thread will still wait, up to 200 seconds (more than 3 minutes !) for this request to appear, before it gives up, closes that connection, and goes back to the pool of available threads. A normal client would not do that, but a badly-intentioned client that tries to create a DOS attack on your server, will do that, just to block threads on your server until there are no more threads available. In this case, it is even worse, because other parameters are using this same value as a default, like : keepAliveTimeout. This one means that once one request has been processed by this Tomcat thread on that connection, the thread will not close this connection, but wait to see if any other request appears on that connection from the same client, within the timeout given. In this case, because it defaults to connectionTimeout, the thread will wait more than 3 minutes to see if there is another request. If the client (browser) does not send any additional request on that connection, you have a thread that is blocked doing nothing, for more than 3 minutes. In this case, the client is in reality the mod_jk module under Apache. So the settings of Apache and mod_jk will matter, and I cannot tell exactly at this point what will happen. But if this was a HTTP Connector open to the external WWW world, what would certainly heppen is that you would run out of threads in Tomcat within a couple of minutes, and your server would stop accepting new requests. Or to put this another way : if this was a HTTP connector open to the outside world, and I knew the address of your server, I could bring it down in 3 seconds, using the standard ab program that comes with Apache. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen Tel.: + 49 (0) 6592 / 712 -2826 Tel.: + 49 (0) 2676 / 9520 -183 Fax: + 49 (0) 6592 / 712 -2829 http://www.telestar.de/ -Ursprüngliche Nachricht- Von: André Warnier [mailto:a...@ice-sa.com] Gesendet: Freitag, 15. März 2013 10:45 An: Tomcat Users List Betreff: Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open Some notes : - according to the comments in your Apache config file, you are using the worker MPM. But are you sure ? what does /usr/sbin/apache2ctl -l say ? Compiled in modules: core.c mod_log_config.c mod_logio.c worker.c http_core.c mod_so.c - we are missing the contents of your /etc/apache2/workers.properties file Sry didn't rename properties, see attached.. - we are missing the JkMounts or equivalent that you are doing from Apache to Tomcat. Same to our default website related questions : you have 2 tomcats. Are you doing load-balancing ? or are you just sending some URLs to tomcat1 and other to tomcat2 ? We doing loadbalacning Are you proxying everything to the tomcats, or is the Apache httpd front-end serving some URLs on its own ? if so, what proportion ? Everything goes to the tomcats - how many hits per second/minute/hour (any of them, approx.) is your server handling ? according to our logs yesterday between 8pm and 9pm we had 73667 hits together on both tomcats. - you do not have any specific timeout parameters set in your Tomcat AJP Connectors, which is basically a good thing : better to leave the defaults in place, than to start playing with settings that you do not really understand, and make things worse. But you do set one : connectionTimeout=20. Which sounds extremely high to me. Re: http://tomcat.apache.org/tomcat-7.0-doc/config/ajp.html It means that when a client connects to that Connector, a Tomcat thread will be started to handle this connection; then the thread will wait on the connection, to read the request. If the request does not appear, this thread will still wait, up to 200 seconds (more than 3 minutes !) for this request to appear, before it gives up, closes that connection, and goes back to the pool of available threads. A normal client would not do that, but a badly-intentioned client that tries to create a DOS attack on your server, will do that, just to block threads on your server until there are no more threads available. In this case, it is even worse, because other parameters are using this same value as a default, like : keepAliveTimeout. This one means that once one request has been processed by this Tomcat thread on that connection, the thread will not close this connection, but wait to see if any other request appears on that connection from the same client, within the timeout given. In this case, because it defaults to connectionTimeout, the thread will wait more than 3 minutes to see if there is another request. If the client (browser) does not send any additional request on that connection, you have a thread that is blocked doing nothing, for more than 3 minutes. In this case, the client is in reality the mod_jk module under Apache. So the settings of Apache and mod_jk will matter, and I cannot tell exactly at this point what will happen. But if this was a HTTP Connector open to the external WWW world, what would certainly heppen is that you would run out of threads in Tomcat within a couple of minutes, and your server would stop accepting new requests. Or to put this another way : if this was a HTTP connector open to the outside world, and I knew the address of your server, I could bring it down in 3 seconds, using the standard ab program that comes with Apache. that is how our servers are connected together: We have another Webserver (different server) connecting to our Apache - Proxy which is loadbalacning our two tomcats. (Apache and tomcat are together at on server) I will go end reduce the timeout on both worker.properties and server.xml. What will be a value? 30 Seconds? Thanks a lot.. workers.tomcat_home=/opt/tomcat workers.java_home=/usr/lib/jvm/java-6-sun ps=/ worker.list=loadbalancer,status #//siehe https://community.jboss.org/wiki/OptimalModjk12Configuration worker.tomcatX.host=localhost worker.tomcatX.type=ajp13 worker.tomcatX.fail_on_status=404 worker.tomcatX.lbfactor=1 worker.tomcatX.ping_timeout=1000 worker.tomcatX.ping_mode=A worker.tomcatX.socket_timeout=10 worker.tomcatX.connection_pool_timeout=200 worker.tomcat1.reference=worker.tomcatX worker.tomcat1.port=7009 worker.tomcat2.reference=worker.tomcatX worker.tomcat2.port=9009 worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=tomcat1,tomcat2 worker.status.type=status VirtualHost *:8080 ServerAdmin webmaster@localhost SetEnvIf User-Agent (internal dummy connection
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 André, On 3/14/13 11:02 AM, André Warnier wrote: But this is the request/response connection, so I doubt that there would be a bug there, otherwise we'd have problems reports filling this list every day. Might there be that there is somewhere a discrepancy between the keep-alive settings, between Apache and Tomcat ? I'd take a look at the connection timeout settings between mod_jk and Tomcat: if they are mismatched, you might get some weirdness where connections are going stale on the Apache side and not the Tomcat side (or vice-versa). It's easy to do since one side expresses timeouts in ms and the other side in seconds. :) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG/MacGPG2 v2.0.17 (Darwin) Comment: GPGTools - http://gpgtools.org Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iEYEAREIAAYFAlFDfBsACgkQ9CaO5/Lv0PD+ggCguhh5KOW+hoxxhwRFIY0AsH8N /30AoJ2sfyMb+xSqewoI0iOSBJRXq0hg =Up57 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: AW:AJP suddenly Stopps acting: ajp on 7009 and 9009 : connections keept open
David Kumar wrote: Hey, thanks for note.. Attached you can find a new list. So, java is keeping these connections in close_wait. Yes. Whether this is the java that runs tomcat is not immediately evident. But it also seems that they concern port numbers like 7009, 9009 etc. which must be your AJP Connectors, so let's assume the CLOSE-WAITs all involve Tomcat. The first observation hat I would make is that this looks like a pretty busy system. Then, there are 2100+ lines in that list. Let's assume first that they are all internal connections (which probable they are not, but for simplicity), so that there are actually 2 lines for the same connection, like these : tcp0 0 127.0.0.1:39872 127.0.0.1:7009 FIN_WAIT2 20766/apache2 ... tcp6 1 0 127.0.0.1:7009 127.0.0.1:39872 CLOSE_WAIT 20564/java .. So let's say that we have in total about 1000 connections active in some state. Compared to that, there are 54 connections in CLOSE_WAIT, which is like 5%. That does not really sound pathological to me, considering that CLOSE_WAIT is a normal state through which any TCP connection goes at some point. If you look again at the above connection, and if I am not confusing my clients and servers again, it looks here like : - Apache http (PID 20766), as a client, has established a connection to port 7009 of the server Tomcat (PID 20564). That must be an AJP connection, from the mod_jk module of Apache, to the AJP Connector in Tomcat. - A while later, Apache closes the connection. It sends a FIN packet to Tomcat, and waits for Tomcat to acknowledge this FIN packet. - when it receives the FIN packet from Apache, Tomcat responds with an ACK, and then goes to the CLOSE_WAIT state. - when Apache receives the ACK from Tomcat, it goes to state FIN_WAIT2. (and that is the current state of the above connection) - now Tomcat is supposed to send a FIN to Apache, and receive an ACK in return. - if Apache receives the FIN from Tomcat, it sends back an ACK, and goes into state TIME_WAIT (where it doesn't expect anything anymore), and then after a short moment, it discards that connection. If there are connections that remain in the CLOSE_WAIT state for very long, it means that something in the last 2 steps above is not working. Now in your listing, we see about 1900 lines in the TIME_WAIT state, all of them *to* an AJP port of Tomcat. This seems to suggest that the Apache side is working as it should. But if the Tomcat side stays in CLOSE_WAIT for a long time, then it would suggest that on the Tomcat side, that connection is never properly close()'d. But this is the request/response connection, so I doubt that there would be a bug there, otherwise we'd have problems reports filling this list every day. Might there be that there is somewhere a discrepancy between the keep-alive settings, between Apache and Tomcat ? or maybe your server is just not strong enough for the load you are putting on it ? How about a netstat -t -pan ? it's a bit easier to find the relevant connections. If needed, tell us which processes are Tomcat instances/threads and which are Apache httpd (if it is not evident from the list). - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Mit freundlichen Grüßen David Kumar Softwareentwickler, B. Sc. Abteilung Infotech - Interaktiv TELESTAR-DIGITAL GmbH Am Weiher 14 D-56766 Ulmen http://www.telestar.de/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
