Re: File descriptors peaks with latest stable build of Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Frederik, On 4/22/15 10:53 AM, Frederik Nosi wrote: Hi, On 04/22/2015 04:35 PM, Christopher Schultz wrote: Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. You can tune the network stack's timeout using sysctl, eg: net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_fin_timeout = 3 This won't do anything, either. As far as the OS is concerned, the application (Tomcat) is still using that connection. Therefore it can't be cleaned up. Tomcat has to actively hang up the connection, and the best way to do that is with synchronized timeouts between the reverse proxy and Tomcat. You can try all other kinds of tricks, but the fact of the matter is that the application is still trying to use the socket, so no other component can step-in and kill it. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7sZAAoJEBzwKT+lPKRYRhkP/j0GBtPH/+/QU2YEgZxbRoJE z2lmWxDrbFNxiYFS5332SvN4bXhG/Khog83CeBM0bg0VLciSxKYqm5J8YziMlrlo omqk3gUiNeViyjsjO5SBW9hxT1qhC1PLdtx7uZ7xUiNmmE24wQ3Gi2edyjyvYDJ0 pzLT+bEp8BjXgm0c6aOONO0PJ+PbyZPeF56PXq6iqn426IhebEUlDP8kxuSh3RwL LQW7tg05bg3yTuP1ZjiwH4gmBfbomJ+xpY6F+zwDkZgk7Cs4okp5/Tr0uTNhsHQM lgGaIZc9SCoqKaMFqWila3RaAnnpqDe1cdg2N44zluIaMkcO94kDSWBuT25t5dGe GBiFG2HGczwyo5MCrx0RgYgLtb2bQ0QZQ8nHzNis8wkNQdHWzziWsvsVQOCnCqL/ 3FOkWUbbJTdmnB8lx84sRcuMsDYQ0BYOYW4W/F2WmSxzBnm7V4NixHG9dD4lZ3vJ fhIO/d0VNOpI+wesZyQg+pwWRHInbigZ0+5A3InOLHW84rWa2qX0wvt6a7rBb0YP gonBY4xbrPTHoXDHH7ZCs3JW+gwstA5avA/Obp45C5LessbduqRPtBvMUZizyZR5 ByPtJcrCvHlFux1fwc7Idj/9seqaYvllyvO6evvhqgYVU3jV2tekOUNuFGDJ8KRt HmrzuiH3cmU1JpT6FSen =XyQw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Hi Andre, If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. In order to do that, these startup params might be necessary in the Java app side : -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.local.only=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false Thanks, Neill On Wed, Apr 22, 2015 at 3:02 PM, André Warnier a...@ice-sa.com wrote: Rainer Jung wrote: Am 22.04.2015 um 11:58 schrieb Thomas Boniface: What concerns me the most is the CLOSE_WAIT on tomcat side because when an fd peak appears the web application appears to be stuck. It feels like all its connections are consumed and none can be established from nginx anymore. Shouldn't the CLOSE_WAIT connection be recycled to received new connections from nginx ? Just to clarify: Every connection has two ends. In netstat the local end is left, the remote end is right. If a connection is between processes both on the same system, it will be shown in netstat twice. Once for each endpoint being the local side. CLOSE_WAIT for a connection between a (local) and b (remote) means, that b has closed the connection but not a. There is no automatism for a closing it because b has closed it. If CLOSE_WAIT pile up, then the idea of b and a when a connection should no longer be used are disparate. E.g. they might have very different idle timeouts (Keep Alive Timeout on HTTP speak), or one observed a problem that the other didn't observe. When I did the counting for Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT the 127.0.0.1:8080 was left in netstat output, so local. It means the other side (whatever is the other side of the connection, likely nginx) has closed the connection alardy, but not Tomcat. And the total number of those connections: Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT 1650127.0.0.1:8080 ESTABLISHED indeed sums up to the default maxConnections 1 mentioned by Chris. What I do not understand is, that the same connections looked at from nginx being the local end, show a totally different statistics: Count IP:Port ConnectionState 20119127.0.0.1:8080 SYN_SENT 4692127.0.0.1:8080 ESTABLISHED 488127.0.0.1:8080 FIN_WAIT2 122127.0.0.1:8080 TIME_WAIT 13127.0.0.1:8080 FIN_WAIT1 But maybe that's a problem to solve after you fixed the CLOSED_WAIT (or the 1000 limit) and redo the whole observation. Pretty big numbers you habe ... Thomas, to elaborate on what Rainer is writing above : A TCP connection consists of 2 pipes, one in each direction (client to server, server to client). From a TCP point of view, the client is the one which initially requests the connection. The server is the one which accepts that connection. (This is different from the more general idea of server, as in Tomcat server. When Tomcat accepts a HTTP connection, it acts as server; when a Tomcat webapp establishes a connection with an external HTTP server, the webapp (and by extension Tomcat) is the client). These 2 pipes can be closed independently of one another, but both need to be closed for the connection to be considered as closed and able to disappear. When the client wants to close the connection, it will send a close request packet on the client-to-server pipe. The server receives this, and knows then that the client will not send anything anymore onto that pipe. For a server application reading that pipe, this would result in the equivalent of an end of file on that datastream. In response to the client close request, the server is supposed to react by not sending any more data onto the server-to-client pipe, and in turn to send a close request onto that pipe. Once these various close messages have been received and acknowledged by both sides of the connection, the connection is considered as closed, and the resources associated with it can be reclaimed/recycled/garbage collected etc.. (closed is like a virtual state; it means that there is no connection). But if one side fails to fulfill its part of that contract, the connection is still there, and it just remains there forever until something forceful terminates it. And all the resources tied to that connection also remain tied to it, and are subtracted from the overall resources which the server has available to perform other tasks. From a server point of view, the ideal situation is when all connections are actually active and really being used to do something useful (sending or receiving data e.g.). The worst situation is when there are many useless connections : connections in some state or the other, not actually doing anything useful, but tying up resources nevertheless.
Re: File descriptors peaks with latest stable build of Tomcat 7
Rainer Jung wrote: Am 22.04.2015 um 11:58 schrieb Thomas Boniface: What concerns me the most is the CLOSE_WAIT on tomcat side because when an fd peak appears the web application appears to be stuck. It feels like all its connections are consumed and none can be established from nginx anymore. Shouldn't the CLOSE_WAIT connection be recycled to received new connections from nginx ? Just to clarify: Every connection has two ends. In netstat the local end is left, the remote end is right. If a connection is between processes both on the same system, it will be shown in netstat twice. Once for each endpoint being the local side. CLOSE_WAIT for a connection between a (local) and b (remote) means, that b has closed the connection but not a. There is no automatism for a closing it because b has closed it. If CLOSE_WAIT pile up, then the idea of b and a when a connection should no longer be used are disparate. E.g. they might have very different idle timeouts (Keep Alive Timeout on HTTP speak), or one observed a problem that the other didn't observe. When I did the counting for Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT the 127.0.0.1:8080 was left in netstat output, so local. It means the other side (whatever is the other side of the connection, likely nginx) has closed the connection alardy, but not Tomcat. And the total number of those connections: Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT 1650127.0.0.1:8080 ESTABLISHED indeed sums up to the default maxConnections 1 mentioned by Chris. What I do not understand is, that the same connections looked at from nginx being the local end, show a totally different statistics: Count IP:Port ConnectionState 20119127.0.0.1:8080 SYN_SENT 4692127.0.0.1:8080 ESTABLISHED 488127.0.0.1:8080 FIN_WAIT2 122127.0.0.1:8080 TIME_WAIT 13127.0.0.1:8080 FIN_WAIT1 But maybe that's a problem to solve after you fixed the CLOSED_WAIT (or the 1000 limit) and redo the whole observation. Pretty big numbers you habe ... Thomas, to elaborate on what Rainer is writing above : A TCP connection consists of 2 pipes, one in each direction (client to server, server to client). From a TCP point of view, the client is the one which initially requests the connection. The server is the one which accepts that connection. (This is different from the more general idea of server, as in Tomcat server. When Tomcat accepts a HTTP connection, it acts as server; when a Tomcat webapp establishes a connection with an external HTTP server, the webapp (and by extension Tomcat) is the client). These 2 pipes can be closed independently of one another, but both need to be closed for the connection to be considered as closed and able to disappear. When the client wants to close the connection, it will send a close request packet on the client-to-server pipe. The server receives this, and knows then that the client will not send anything anymore onto that pipe. For a server application reading that pipe, this would result in the equivalent of an end of file on that datastream. In response to the client close request, the server is supposed to react by not sending any more data onto the server-to-client pipe, and in turn to send a close request onto that pipe. Once these various close messages have been received and acknowledged by both sides of the connection, the connection is considered as closed, and the resources associated with it can be reclaimed/recycled/garbage collected etc.. (closed is like a virtual state; it means that there is no connection). But if one side fails to fulfill its part of that contract, the connection is still there, and it just remains there forever until something forceful terminates it. And all the resources tied to that connection also remain tied to it, and are subtracted from the overall resources which the server has available to perform other tasks. From a server point of view, the ideal situation is when all connections are actually active and really being used to do something useful (sending or receiving data e.g.). The worst situation is when there are many useless connections : connections in some state or the other, not actually doing anything useful, but tying up resources nevertheless. This can get to the point where some inherent limit is reached, and the server cannot accept any more connections, although in theory it still has enough other resources available which would allow it to process more useful transactions. Most of the TCP states that you see in the netstat output are transient, and last only a few milliseconds usually. They are just part of the overall TCP connection lifecycle which is cast in stone and which you can do nothing about. But, for example, if there is a permanent very high number of connections in the CLOSE_WAIT state, that is not normal. See here
Re: Apache Tomcat jk connector 400 bad request
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/21/15 5:18 PM, Razi wrote: Another bit of information I wanted to add. The apache error log is peppered with the following line : OpenSSL : I/O error, 5 bytes expected to read on BIO#... Are those errors correlated with the AJAX failures? Do non-AJAX requests fail in this way? Please copy/paste your Tomcat Connector configuration as well as your mod_jk properties for this worker. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7RlAAoJEBzwKT+lPKRY9swP/3zVEi9HE2Y5LshvuasAsk1R FiSty8XMgcsNcdM7DNG44TWHlxzaI8NjSxeoVbQAY1ENjGyz7PhkiRiD4CLBXV70 /DnlILOy/M5vT+O7pcdW0tHsF9XmeewkcN1VAef3fp3f+NbQU6U1xXpwDp+yWETy 7jOB0yBA/aqJPwkxqdfHd4BCi0+BvJBgUoSJ3vo67oYSA5lo0u/42zRWRgD1513k u14EJuYQfcevkwUMiX75KehvBqSj8O31zoYOBbeUjQSq2qshpJ+RKnDny1ZibHT5 ZfbtPEP9lvZChS/qdiz+vqHVQqPp67qnkDf4aO+TehSNyRfJ/vzMP93fwELWa4dB /AqAvOpxhN0bq9NN6vHLce5TY/5b6hhTGHuPqR8zXCS0VS8lgKGqkCJ5NmJfkAw/ Vdx0y60Uad01fCbSipT+/8X9zRpvOiaxvqix1s8NuiOHFdjDXEdnck0JdkWq4Wnm 2RfhZ9u9WixDEaS2lUQabSmHYxbB2gQBaL0kWQdttwaEKIaoXIry3IcEAreS8C9K ELloPXCR1qQXAv71Up31VYkK2jXmTE8QMIKrJOcd2hZPAQExKvVur2jTp/TFRa30 a2Qr2Gebnv9qwdJoM3bVTshgEROOMIMicOAGSn1juwOQ478u9TA2zKDJIALP7VyG 9/7QwAFFNxHMjghzzVXy =mWJG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Hello Christopher S., I know it won't. I just wanted to provide insight into Andre W.'s approach. Thanks, Neill On Wed, Apr 22, 2015 at 4:58 PM, André Warnier a...@ice-sa.com wrote: Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. Just humor me.. If it doesn't, it doesn't. But it's easy to do, does not require a change of configuration nor a shutdown/restart of Tomcat, and it may show us something in principle unexpected. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
On 04/22/2015 05:15 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Frederik, On 4/22/15 10:53 AM, Frederik Nosi wrote: Hi, On 04/22/2015 04:35 PM, Christopher Schultz wrote: Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. You can tune the network stack's timeout using sysctl, eg: net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_fin_timeout = 3 This won't do anything, either. As far as the OS is concerned, the application (Tomcat) is still using that connection. Therefore it can't be cleaned up. Indeed you are right, tuning the network stack help with TIME_WAIT, not CLOSE_WAIT, my bad. Tomcat has to actively hang up the connection, and the best way to do that is with synchronized timeouts between the reverse proxy and Tomcat. You can try all other kinds of tricks, but the fact of the matter is that the application is still trying to use the socket, so no other component can step-in and kill it. Probably the application is slow then, or the server overloaded. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7sZAAoJEBzwKT+lPKRYRhkP/j0GBtPH/+/QU2YEgZxbRoJE z2lmWxDrbFNxiYFS5332SvN4bXhG/Khog83CeBM0bg0VLciSxKYqm5J8YziMlrlo omqk3gUiNeViyjsjO5SBW9hxT1qhC1PLdtx7uZ7xUiNmmE24wQ3Gi2edyjyvYDJ0 pzLT+bEp8BjXgm0c6aOONO0PJ+PbyZPeF56PXq6iqn426IhebEUlDP8kxuSh3RwL LQW7tg05bg3yTuP1ZjiwH4gmBfbomJ+xpY6F+zwDkZgk7Cs4okp5/Tr0uTNhsHQM lgGaIZc9SCoqKaMFqWila3RaAnnpqDe1cdg2N44zluIaMkcO94kDSWBuT25t5dGe GBiFG2HGczwyo5MCrx0RgYgLtb2bQ0QZQ8nHzNis8wkNQdHWzziWsvsVQOCnCqL/ 3FOkWUbbJTdmnB8lx84sRcuMsDYQ0BYOYW4W/F2WmSxzBnm7V4NixHG9dD4lZ3vJ fhIO/d0VNOpI+wesZyQg+pwWRHInbigZ0+5A3InOLHW84rWa2qX0wvt6a7rBb0YP gonBY4xbrPTHoXDHH7ZCs3JW+gwstA5avA/Obp45C5LessbduqRPtBvMUZizyZR5 ByPtJcrCvHlFux1fwc7Idj/9seqaYvllyvO6evvhqgYVU3jV2tekOUNuFGDJ8KRt HmrzuiH3cmU1JpT6FSen =XyQw -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Hi, On 04/22/2015 04:35 PM, Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. You can tune the network stack's timeout using sysctl, eg: net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1 net.ipv4.tcp_fin_timeout = 3 - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7GvAAoJEBzwKT+lPKRYPUIP+gIbeAUIJNo7budTt59iriqX JJzpceyz8RQUPmqOKpfSSj+5xLL9wFUBe9WaNsTLgdGkTxPWk1O/UYqmOmeYjYm6 tKkL/VwI6ySghkKImIMBAOpa8up6cvjqRbziu6He0K7gMgf1d8ipcPI0GQdmGWlm 7sMM9FWgQiBtP1+WrRFyEH/7ldD3xbGfgrdzYO4RAqaqbtplvS8ept8ecXZp12RT RUeUIZByHBE2x39mcN2piZkPtAB0htN/DPSuAAPi850bBo5cECLlbyDusEoWa4G/ LQX6i5iIe68M6u2HqRM2gGPB/5LxDnBrCbQdVpcyGBr0CbI/NcfpxKx5IQYYf7PP fG5RV3EViqLuIuMTzlMig1b/6h0djGCmMZc8JIZWlsX1SQXf/gbHPOIwEBE2M4pD wtfoXZjWOmPep+a8y5QbiiYGZo5wIp9dKNdZEta4KIa/WAUkIYwVT5dEQS9pZ7N0 /M4NRDngbPdL7FZNh2q4/FNm/gR9W7bg5iIpjz5wVpEwhvqpjU7kJ/rIIE1Vdh6/ VbRI72dE3P9W1qm8XeQwGFkv8uHcTCqRVxPVN5JvQTIkbWF54tUqvFFB4Dk4gHYW s6yDRQdHIAGRnAH9nSF4xp4Bdl9vhl/zAoEW44/MUdeeuknxROMPBerHG3QwoMFw MSjyDI5wdBDuFCcuex+s =OINr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org Cheers, Frederik - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Christopher Schultz wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. Just humor me.. If it doesn't, it doesn't. But it's easy to do, does not require a change of configuration nor a shutdown/restart of Tomcat, and it may show us something in principle unexpected. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Getting Apache 2.2.29 to work with tomcat 8.0.21 using mod_jk 1.2.40 on a mac
Thanks a lot *Neven *and* Rainer*!! That was my problem!! I changed the worker port to 8009 and everything is good. Thanks for your help!! On Wed, Apr 22, 2015 at 4:46 AM, Rainer Jung rainer.j...@kippdata.de wrote: Am 22.04.2015 um 11:33 schrieb Mayuresh: Hi Gurus, i am trying to make this combination and am stumbled at a place. After a lot of searching and experimenting I finally decided to post this question here. I am trying to load a page which loads just fine directly on tomcat. However if I try to load it through the Apache web server, it fails with a 502 on apache (400 on tomcat) For an unsuccessful request through apache I see the following in the logs, where all the headers seem to be garbled: 22-Apr-2015 01:53:46.656 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@616702aa Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:53:46.657 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [^R4^B^G^B^B^@^HHTTP/1.1^@^@^M/greeting/abc^@^@ 10.0.0.23^@ÿÿ^@^Posxltmkshi.local^@ü^@^@ ^K^@^Uosxltmkshi.local:9980^@ ^F^@ keep-alive^@ ^L^@^Hno-cache^@^@^MCache-Control^@^@^Hno-cache^@ ^A^@Jtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^@ ^N^@wMozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^@^@^OAccept-Encoding^@^@^Sgzip, deflate, sdch^@^@^OAccept-Language^@^@^Nen-US,en;q=0.8^@^@^OX-Forwarded-For^@^@^L14.102.224.6^@ ^H^@^A0^@ ^@^OAJP_REMOTE_PORT^@^@^E55806^@ ^@^PJK_LB_ACTIVATION^@^@^CACT^@ÿ] mod_jk talks the AJP protocol, so you need to connect it to the Tomcat AJP connector. You instead connected it to the Tomcat HTTP connector. You probably have configured port 8080 in the mod_jk config instead of 8009 (default AJP port in Tomcat default server.xml). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- -Mayuresh
Re: File descriptors peaks with latest stable build of Tomcat 7
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Neill, On 4/22/15 9:12 AM, Neill Lima wrote: If I am not wrong, if the application in question is monitored in VisualVM through JMX (https://visualvm.java.net/) you could trigger a Force GC from its monitoring console. You can do this, but it won't close any CLOSE_WAIT connections. Tomcat's timeout must be reached. I suspect that the timeout(s) are simply way too long. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7GvAAoJEBzwKT+lPKRYPUIP+gIbeAUIJNo7budTt59iriqX JJzpceyz8RQUPmqOKpfSSj+5xLL9wFUBe9WaNsTLgdGkTxPWk1O/UYqmOmeYjYm6 tKkL/VwI6ySghkKImIMBAOpa8up6cvjqRbziu6He0K7gMgf1d8ipcPI0GQdmGWlm 7sMM9FWgQiBtP1+WrRFyEH/7ldD3xbGfgrdzYO4RAqaqbtplvS8ept8ecXZp12RT RUeUIZByHBE2x39mcN2piZkPtAB0htN/DPSuAAPi850bBo5cECLlbyDusEoWa4G/ LQX6i5iIe68M6u2HqRM2gGPB/5LxDnBrCbQdVpcyGBr0CbI/NcfpxKx5IQYYf7PP fG5RV3EViqLuIuMTzlMig1b/6h0djGCmMZc8JIZWlsX1SQXf/gbHPOIwEBE2M4pD wtfoXZjWOmPep+a8y5QbiiYGZo5wIp9dKNdZEta4KIa/WAUkIYwVT5dEQS9pZ7N0 /M4NRDngbPdL7FZNh2q4/FNm/gR9W7bg5iIpjz5wVpEwhvqpjU7kJ/rIIE1Vdh6/ VbRI72dE3P9W1qm8XeQwGFkv8uHcTCqRVxPVN5JvQTIkbWF54tUqvFFB4Dk4gHYW s6yDRQdHIAGRnAH9nSF4xp4Bdl9vhl/zAoEW44/MUdeeuknxROMPBerHG3QwoMFw MSjyDI5wdBDuFCcuex+s =OINr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat jk connector 400 bad request
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/22/15 6:39 PM, Razi Ansari wrote: Original message From: Christopher Schultz ch...@christopherschultz.net Date: 04/23/2015 6:15 AM (GMT+08:00) To: Tomcat Users List users@tomcat.apache.org Subject: Re: Apache Tomcat jk connector 400 bad request If you decide to change your timeout values (I don't see a reason to do so unless you are encountering some kind of related problem, and I don't consider this to be a related problem), make sure they are consistent between mod_jk and Tomcat's AJP connector. I tried inceasing the value of webserver KeepAliveTimeout to 15 and i dont get the 400 bad request error in the test environment. But i really want to know the root cause before i move it to production. In my httpwatch i see the request going without any body ony header. Without the change in KeepAliveTimeout, can you reproduce this error in your test environment? I would honestly be surprised if KeepAliveTimeout is the trouble, here. For the AJP connector, the KeepAliveTimeout is not relevant because AJP is by definition keep-alive. You can disable it if you think doing so will improve the situation, but I don't have any reason to suspect it would. It would be great to know whether the AJP connector or the HTTP connector was the one failing. In Tomcat, the stack trace will include the thread identifier which includes the port number and protocol being used. Please include that with your stack traces when you find the m. It would also be great to know which (exact) version of Tomcat is being used under the hood. Check the logs during startup to see what Tomcat says it is. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVODbWAAoJEBzwKT+lPKRYiBkP+QENinLAv9vXdjDi598Av56n nin9n4jzkgVaMC7h/EpzM81w7rjVoNdfD9d3j472hQFyUY+x9iQwDxNRc25t7VHH jwq21AZBqRDn6hNvLxxpXyKqVuLImZ08GFzI4PdelIQ0IuZ2WlRQwW1xNtcVZKgh yq1/Az1zK82rzrIH8WJhu0frYNd5mqrrEEVCQfUs3pEJyqf0uQFM8buVVKScla3x sb6qjo3+XCHGZ/KXFllr0t0E7fGlk7xP8NhRLzSO7AkDvUgk7SzAryN2VmwbTn3w KbO3bzT7dQse+1ykj246L+TqIWJOsycooGagTRtH7kW/5jRXS6b8kLL08XhLY70z Ybcfbd1gwIbihm29LsOaU54hWTyQYa5cSAzCXEaQsGUfaoDsuU4L69Uf+G40QcO6 c1cCvJImSFjpmVcoVtVitT+U6vDDK0kIcuyrb766GCyde2pRlFswCMho2GN2yT6L 2e9v9xqmLANZE0iaJpK2qUVE/efh1m+API/wFYXqnQhe2S5V8fXAGXM/rJdeOQNL IXMZN4E1moVClbaKVn+TbtcexR9nmU6GPyaL+VoJauDJf5UV5Y31+wp+XgP1k13/ wQ0LoumpioPYp7QEbrSgNTUF5I2Q+Cp7VGtoO3PPDWzgcBH8eSiqHVnsC2da51k8 sjaBXvJn5/taUN49pEGo =0W9M -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat jk connector 400 bad request
Christopher, When KeepAliveTimeOut is 5, 12 , I get the error, when its 15 and above I don't get the error. The versions of Apahce and Mod_jk are as follows Apache 2.2.24 Mod_jk 1.2.37 I dont use Tomcat, my applicaton server is JBOSS EAP 6.1.0 I will get back to you with more logs. Thanks Warm Regards Razi A. Ansari -Original Message- From: Christopher Schultz Sent: Thursday, April 23, 2015 8:03 AM To: Tomcat Users List Subject: Re: Apache Tomcat jk connector 400 bad request -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/22/15 6:39 PM, Razi Ansari wrote: Original message From: Christopher Schultz ch...@christopherschultz.net Date: 04/23/2015 6:15 AM (GMT+08:00) To: Tomcat Users List users@tomcat.apache.org Subject: Re: Apache Tomcat jk connector 400 bad request If you decide to change your timeout values (I don't see a reason to do so unless you are encountering some kind of related problem, and I don't consider this to be a related problem), make sure they are consistent between mod_jk and Tomcat's AJP connector. I tried inceasing the value of webserver KeepAliveTimeout to 15 and i dont get the 400 bad request error in the test environment. But i really want to know the root cause before i move it to production. In my httpwatch i see the request going without any body ony header. Without the change in KeepAliveTimeout, can you reproduce this error in your test environment? I would honestly be surprised if KeepAliveTimeout is the trouble, here. For the AJP connector, the KeepAliveTimeout is not relevant because AJP is by definition keep-alive. You can disable it if you think doing so will improve the situation, but I don't have any reason to suspect it would. It would be great to know whether the AJP connector or the HTTP connector was the one failing. In Tomcat, the stack trace will include the thread identifier which includes the port number and protocol being used. Please include that with your stack traces when you find the m. It would also be great to know which (exact) version of Tomcat is being used under the hood. Check the logs during startup to see what Tomcat says it is. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVODbWAAoJEBzwKT+lPKRYiBkP+QENinLAv9vXdjDi598Av56n nin9n4jzkgVaMC7h/EpzM81w7rjVoNdfD9d3j472hQFyUY+x9iQwDxNRc25t7VHH jwq21AZBqRDn6hNvLxxpXyKqVuLImZ08GFzI4PdelIQ0IuZ2WlRQwW1xNtcVZKgh yq1/Az1zK82rzrIH8WJhu0frYNd5mqrrEEVCQfUs3pEJyqf0uQFM8buVVKScla3x sb6qjo3+XCHGZ/KXFllr0t0E7fGlk7xP8NhRLzSO7AkDvUgk7SzAryN2VmwbTn3w KbO3bzT7dQse+1ykj246L+TqIWJOsycooGagTRtH7kW/5jRXS6b8kLL08XhLY70z Ybcfbd1gwIbihm29LsOaU54hWTyQYa5cSAzCXEaQsGUfaoDsuU4L69Uf+G40QcO6 c1cCvJImSFjpmVcoVtVitT+U6vDDK0kIcuyrb766GCyde2pRlFswCMho2GN2yT6L 2e9v9xqmLANZE0iaJpK2qUVE/efh1m+API/wFYXqnQhe2S5V8fXAGXM/rJdeOQNL IXMZN4E1moVClbaKVn+TbtcexR9nmU6GPyaL+VoJauDJf5UV5Y31+wp+XgP1k13/ wQ0LoumpioPYp7QEbrSgNTUF5I2Q+Cp7VGtoO3PPDWzgcBH8eSiqHVnsC2da51k8 sjaBXvJn5/taUN49pEGo =0W9M -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Apache Tomcat jk connector 400 bad request
Please check inline for my reply . Thanks. Original message From: Christopher Schultz ch...@christopherschultz.net Date: 04/23/2015 6:15 AM (GMT+08:00) To: Tomcat Users List users@tomcat.apache.org Subject: Re: Apache Tomcat jk connector 400 bad request -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/22/15 5:45 PM, Razi wrote: Hi Christopher, Thanks for looking into this. Any random request fails, ajax or non-ajax. The worker.properties is as follows:: worker.lbroutex.type=lb worker.lbroutex.balance_workers=workerx,workery,workerz worker.lbroute.sticky_session=1 worker.workerX.port=1234 worker.workerX.host=$$$.com worker.workerX.type=ajp13 worker.workerX.lbfactor=1 As I am using jboss EAP, I am pasting the config from my standalone.xml subsystem xmlns=urn:jboss:domain:web:1.4 default-virtual-server=default-host instance-id=worker10 native=false connector name=http scheme=http protocol=HTTP/1.1 socket-binding=http/ connector name=https scheme=http protocol=HTTP/1.1 socket-binding=https secure=true ssl name=ssl key-alias=jboss password= certificate-key-file= protocol=/ /connector connector name=ajp scheme=http protocol=AJP/1.3 socket-binding=ajp/ I suppose this has a default port number (8009?) and it matches what you have in worker.workerX.port? Yes You shouldn't be using any OpenSSL for the the AJP connector, so proxied requests via AJP shouldn't trigger the OpenSSL errors. A full stack trace would be very helpful. I wiill get this trace out I don't see anything immediately obvious. Do you have a firewall between httpd and Tomcat? Has it been configured to leave the connections open forever? If not, you might want to consider configuring CPING/CPONG at intervals (look at the AJP connector configuration reference and search for cping/cpong to see how to do that ). will check for the firewall and revert. If you decide to change your timeout values (I don't see a reason to do so unless you are encountering some kind of related problem, and I don't consider this to be a related problem), make sure they are consistent between mod_jk and Tomcat's AJP connector.I tried inceasing the value of webserver KeepAliveTimeout to 15 and i dont get the 400 bad request error in the test environment. But i really want to know the root cause before i move it to production. In my httpwatch i see the request going without any body ony header. Any chance the 400 responses always come from the same httpd instance or Tomcat instance? No its random any instance. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVOB1GAAoJEBzwKT+lPKRY51AQAJCuO+cuurN9CshykgXm+M2V P+oSLO3wCRrF8WhJtzDz/CfIGT6T679lQSAbMBNliVrxuu+Q+/UbAS4rHcAkO2Ou GRCb1xCX1qoSpl5qmppFsJMMMAZE7NhWP3ZkCC/FdRyj+Lb4ZJcuKmx8LaIEfuvF akWUhJD1sYAhclyYInpF78kXOyBcuP1/6dOtXtlUqZ0JuZiDGHEgdywVRJiZjzpz aTxSdz0AkL4/7svPBn5I6foV2vLUKindQjSn2L1Mjq5TsbEo3Dhe9xxSCL7dmK8u ZJ7Wp8Hi7Z3NrVMNf4YnLiChKhWN9mIhucMQMsY9nez7h2GO2P0LrVo5XRJV5R65 gzagTK1qSHjKrGJqvuBY757j3rsnG9jNEPVixg1IqIA/JIuC/CfO+eVsVsUJAQSH 3KuF3Ata098fBQAMxb9O4vegBm/JRjwJYeKXb+b+fjDIr6QFFgUBv8DNv7cultG9 zDVfZQ2vJqb+TBz5kf8gI5G0ZcwkL2+WveUt+GzvRRbAXiyTmCwIoMGDesi5VIx5 ojUcRWccXjKj6Cg3DKNqWw137/Cre+xVKgepkTjpPOQ0dyAK3G2cUuCgZhJy3OQj FPtfg4vXS5BJQLZCNVWktkLGaQqRwHTIeM7EAwvbaDh+290fZcg1vW99zqCGCLgt ksfiwrTNMLcEDPX//PrL =KsRp -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Info: WebSocket connection closed, Code: 1006 when try to run websocket sample
Hi, I've embedded tomcat 8.0.20 and using that. I build the tomcat websocket example and when I'm trying to run it in my server for all 4 examples (Echo, chat, etc..) I'm getting Info: WebSocket connection closed, Code: 1006 In the console it prints WebSocket connection to 'ws:// 10.100.7.26:9763/examples/websocket/echoProgrammatic' failed: Error during WebSocket handshake: Unexpected response code: 302 This same war file is running perfectly on Tomcat distribution. Can someone tell me what could be the reason for this? In the pom, my dependency is as follows dependency groupIdorg.apache.tomcat/groupId artifactIdtomcat-websocket/artifactId version8.0.20/version /dependency Thanks Best Regards --
Re: Apache Tomcat jk connector 400 bad request
Hi Christopher, Thanks for looking into this. Any random request fails, ajax or non-ajax. The worker.properties is as follows:: worker.lbroutex.type=lb worker.lbroutex.balance_workers=workerx,workery,workerz worker.lbroute.sticky_session=1 worker.workerX.port=1234 worker.workerX.host=$$$.com worker.workerX.type=ajp13 worker.workerX.lbfactor=1 As I am using jboss EAP, I am pasting the config from my standalone.xml subsystem xmlns=urn:jboss:domain:web:1.4 default-virtual-server=default-host instance-id=worker10 native=false connector name=http scheme=http protocol=HTTP/1.1 socket-binding=http/ connector name=https scheme=http protocol=HTTP/1.1 socket-binding=https secure=true ssl name=ssl key-alias=jboss password= certificate-key-file= protocol=/ /connector connector name=ajp scheme=http protocol=AJP/1.3 socket-binding=ajp/ virtual-server name=default-host enable-welcome-root=true alias name=localhost / alias name=example.com / /virtual-server /subsystem Warm Regards Razi A. Ansari -Original Message- From: Christopher Schultz Sent: Wednesday, April 22, 2015 10:47 PM To: Tomcat Users List Subject: Re: Apache Tomcat jk connector 400 bad request -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/21/15 5:18 PM, Razi wrote: Another bit of information I wanted to add. The apache error log is peppered with the following line : OpenSSL : I/O error, 5 bytes expected to read on BIO#... Are those errors correlated with the AJAX failures? Do non-AJAX requests fail in this way? Please copy/paste your Tomcat Connector configuration as well as your mod_jk properties for this worker. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVN7RlAAoJEBzwKT+lPKRY9swP/3zVEi9HE2Y5LshvuasAsk1R FiSty8XMgcsNcdM7DNG44TWHlxzaI8NjSxeoVbQAY1ENjGyz7PhkiRiD4CLBXV70 /DnlILOy/M5vT+O7pcdW0tHsF9XmeewkcN1VAef3fp3f+NbQU6U1xXpwDp+yWETy 7jOB0yBA/aqJPwkxqdfHd4BCi0+BvJBgUoSJ3vo67oYSA5lo0u/42zRWRgD1513k u14EJuYQfcevkwUMiX75KehvBqSj8O31zoYOBbeUjQSq2qshpJ+RKnDny1ZibHT5 ZfbtPEP9lvZChS/qdiz+vqHVQqPp67qnkDf4aO+TehSNyRfJ/vzMP93fwELWa4dB /AqAvOpxhN0bq9NN6vHLce5TY/5b6hhTGHuPqR8zXCS0VS8lgKGqkCJ5NmJfkAw/ Vdx0y60Uad01fCbSipT+/8X9zRpvOiaxvqix1s8NuiOHFdjDXEdnck0JdkWq4Wnm 2RfhZ9u9WixDEaS2lUQabSmHYxbB2gQBaL0kWQdttwaEKIaoXIry3IcEAreS8C9K ELloPXCR1qQXAv71Up31VYkK2jXmTE8QMIKrJOcd2hZPAQExKvVur2jTp/TFRa30 a2Qr2Gebnv9qwdJoM3bVTshgEROOMIMicOAGSn1juwOQ478u9TA2zKDJIALP7VyG 9/7QwAFFNxHMjghzzVXy =mWJG -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat jk connector 400 bad request
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Razi, On 4/22/15 5:45 PM, Razi wrote: Hi Christopher, Thanks for looking into this. Any random request fails, ajax or non-ajax. The worker.properties is as follows:: worker.lbroutex.type=lb worker.lbroutex.balance_workers=workerx,workery,workerz worker.lbroute.sticky_session=1 worker.workerX.port=1234 worker.workerX.host=$$$.com worker.workerX.type=ajp13 worker.workerX.lbfactor=1 As I am using jboss EAP, I am pasting the config from my standalone.xml subsystem xmlns=urn:jboss:domain:web:1.4 default-virtual-server=default-host instance-id=worker10 native=false connector name=http scheme=http protocol=HTTP/1.1 socket-binding=http/ connector name=https scheme=http protocol=HTTP/1.1 socket-binding=https secure=true ssl name=ssl key-alias=jboss password= certificate-key-file= protocol=/ /connector connector name=ajp scheme=http protocol=AJP/1.3 socket-binding=ajp/ I suppose this has a default port number (8009?) and it matches what you have in worker.workerX.port? You shouldn't be using any OpenSSL for the the AJP connector, so proxied requests via AJP shouldn't trigger the OpenSSL errors. A full stack trace would be very helpful. I don't see anything immediately obvious. Do you have a firewall between httpd and Tomcat? Has it been configured to leave the connections open forever? If not, you might want to consider configuring CPING/CPONG at intervals (look at the AJP connector configuration reference and search for cping/cpong to see how to do that ). If you decide to change your timeout values (I don't see a reason to do so unless you are encountering some kind of related problem, and I don't consider this to be a related problem), make sure they are consistent between mod_jk and Tomcat's AJP connector. Any chance the 400 responses always come from the same httpd instance or Tomcat instance? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v2 Comment: GPGTools - http://gpgtools.org iQIcBAEBCAAGBQJVOB1GAAoJEBzwKT+lPKRY51AQAJCuO+cuurN9CshykgXm+M2V P+oSLO3wCRrF8WhJtzDz/CfIGT6T679lQSAbMBNliVrxuu+Q+/UbAS4rHcAkO2Ou GRCb1xCX1qoSpl5qmppFsJMMMAZE7NhWP3ZkCC/FdRyj+Lb4ZJcuKmx8LaIEfuvF akWUhJD1sYAhclyYInpF78kXOyBcuP1/6dOtXtlUqZ0JuZiDGHEgdywVRJiZjzpz aTxSdz0AkL4/7svPBn5I6foV2vLUKindQjSn2L1Mjq5TsbEo3Dhe9xxSCL7dmK8u ZJ7Wp8Hi7Z3NrVMNf4YnLiChKhWN9mIhucMQMsY9nez7h2GO2P0LrVo5XRJV5R65 gzagTK1qSHjKrGJqvuBY757j3rsnG9jNEPVixg1IqIA/JIuC/CfO+eVsVsUJAQSH 3KuF3Ata098fBQAMxb9O4vegBm/JRjwJYeKXb+b+fjDIr6QFFgUBv8DNv7cultG9 zDVfZQ2vJqb+TBz5kf8gI5G0ZcwkL2+WveUt+GzvRRbAXiyTmCwIoMGDesi5VIx5 ojUcRWccXjKj6Cg3DKNqWw137/Cre+xVKgepkTjpPOQ0dyAK3G2cUuCgZhJy3OQj FPtfg4vXS5BJQLZCNVWktkLGaQqRwHTIeM7EAwvbaDh+290fZcg1vW99zqCGCLgt ksfiwrTNMLcEDPX//PrL =KsRp -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Apache Tomcat jk connector 400 bad request
On Wed, Apr 22, 2015 at 5:18 AM, Razi razi...@hotmail.com wrote: hi there, Another bit of information I wanted to add. The apache error log is peppered with the following line : OpenSSL : I/O error, 5 bytes expected to read on BIO#... Warm Regards Razi A. Ansari From: Razi Sent: Tuesday, April 21, 2015 8:37 AM To: users@tomcat.apache.org Subject: Apache Tomcat jk connector 400 bad request Hi there, I would like to explain my scenario, perhaps this has been answered on this forum. A bunch of random Ajax requests from the browser (IE9) end up with a 400 error code on the apache webserver and the the browser hangs for 5 minutes. Httpwatch shows the error code as ERROR_INTERNET_CONNECTION_RESET and then immediately afterwards IE fires the same request again, which shows up with a time taken of 5 minutes and error code as ERROR_HTTP_INVALID_SERVER_RESPONSE. The browser recovers after 5 minutes. Further investigation on the webserver and appserver logs reveals the following:: a.. The request comes from the browser and hits the webserver and then forwards to the appserver instantly. b.. The mod_jk log for the request shows that there is time duration of 5 minutes spent in the ajp_read_fully_server::jk_ajp_common.c(1399): enter. After 5 minutes I get the next line as follows ajp_read_fully_server::jk_ajp_common.c(1432): exit. Then in the next line i see the following ajp_send_request::jk_ajp_common.c(1766) worker 11 browser stop sending data, no need to recover. Later it shows unrecoverable 400, request failed. c.. The forensic.log show the content length as a nonzero value. d.. The applcation server log hangs in the org.apache.coyote.ajp.AjpProcessor.read method for 5 mintues and the continues the execution. The thread dump also confirms this. The questions I have are:: a.. Is this a problem with IE only because of the keepalive timeout and the apache webserver keepalive time(current value is set to 5seconds) out which is not in sync. b.. Is this a problem with the appserver not able to process requests that are bad/incomplete. c.. Should I increase the Apache webserver timeout value to 60s or more , will this have any performance impact. Kindly advise on the scenario. Many thanks for reading through. Current setup: Apache 2.2.24 Mod_jk 1.2.37 Redhat Linux VM JBoss EAP 6.1.0 JSF 2.1, Richfaces 3.3.4 First, I apologize if my comment is offtopic. But you may also consider fronting nginx instead of Apache httpd. It is very easy to configure as reverse proxy for your tomcat and also serve your static files. If Apache is really a requirement kindly ignore my comment Warm Regards Razi A. Ansari -- Marcos | I love PHP, Linux, and Java http://javadevnotes.com/java-string-length-examples
RE: Apache Tomcat jk connector 400 bad request
Hi Marcos Thanks for the comment. I am using apache to serve static content and it works well. Its just these random requests that fail with 400. I am trying out using a slightly higher value of the KeepAliveTimeout. I wanted to get some advise from the experts in this forum . Thanks. Original message From: Marcos Almeida Azevedo marcos.al.azev...@gmail.com Date: 04/22/2015 3:37 PM (GMT+08:00) To: Tomcat Users List users@tomcat.apache.org Subject: Re: Apache Tomcat jk connector 400 bad request On Wed, Apr 22, 2015 at 5:18 AM, Razi razi...@hotmail.com wrote: hi there, Another bit of information I wanted to add. The apache error log is peppered with the following line : OpenSSL : I/O error, 5 bytes expected to read on BIO#... Warm Regards Razi A. Ansari From: Razi Sent: Tuesday, April 21, 2015 8:37 AM To: users@tomcat.apache.org Subject: Apache Tomcat jk connector 400 bad request Hi there, I would like to explain my scenario, perhaps this has been answered on this forum. A bunch of random Ajax requests from the browser (IE9) end up with a 400 error code on the apache webserver and the the browser hangs for 5 minutes. Httpwatch shows the error code as ERROR_INTERNET_CONNECTION_RESET and then immediately afterwards IE fires the same request again, which shows up with a time taken of 5 minutes and error code as ERROR_HTTP_INVALID_SERVER_RESPONSE. The browser recovers after 5 minutes. Further investigation on the webserver and appserver logs reveals the following:: a.. The request comes from the browser and hits the webserver and then forwards to the appserver instantly. b.. The mod_jk log for the request shows that there is time duration of 5 minutes spent in the ajp_read_fully_server::jk_ajp_common.c(1399): enter. After 5 minutes I get the next line as follows ajp_read_fully_server::jk_ajp_common.c(1432): exit. Then in the next line i see the following ajp_send_request::jk_ajp_common.c(1766) worker 11 browser stop sending data, no need to recover. Later it shows unrecoverable 400, request failed. c.. The forensic.log show the content length as a nonzero value. d.. The applcation server log hangs in the org.apache.coyote.ajp.AjpProcessor.read method for 5 mintues and the continues the execution. The thread dump also confirms this. The questions I have are:: a.. Is this a problem with IE only because of the keepalive timeout and the apache webserver keepalive time(current value is set to 5seconds) out which is not in sync. b.. Is this a problem with the appserver not able to process requests that are bad/incomplete. c.. Should I increase the Apache webserver timeout value to 60s or more , will this have any performance impact. Kindly advise on the scenario. Many thanks for reading through. Current setup: Apache 2.2.24 Mod_jk 1.2.37 Redhat Linux VM JBoss EAP 6.1.0 JSF 2.1, Richfaces 3.3.4 First, I apologize if my comment is offtopic. But you may also consider fronting nginx instead of Apache httpd. It is very easy to configure as reverse proxy for your tomcat and also serve your static files. If Apache is really a requirement kindly ignore my comment Warm Regards Razi A. Ansari -- Marcos | I love PHP, Linux, and Java http://javadevnotes.com/java-string-length-examples
Re: Problem with SimpleTcpCluster and nodes temporarily dropping out of cluster
Hi! Thanks for your reply! We have evaluated backupmanager and it does indeed seem to solve the issue! br /Linus On 8 April 2015 at 12:30, Mark Thomas ma...@apache.org wrote: On 07/04/2015 21:46, Linus Brimstedt wrote: Hello, We have a problem with SimpleTcpCluster and session replication. We have tested this in Tomcat 7 and 8 (latest versions) on Windows Server using latest revision of Hotspot JDK version 7, but checking the code it seems like this is not covered anyway. The problem: Replication works well when servers start up, they get missing sessions and new sessions are replicated to all servers. If one server temporarily drops out of the cluster, for example because it gets unresponsive during deployment of new application, the sessions created during this time are not replicated. We remove servers from load balancer several times a day to upgrade different applications, so this has become a big problem for us. Once the server joins the cluster again, new sessions are replicated properly. Only sessions created during the unresponsive window are lost. I see two solutions: 1) Switch to a different session sharing setup (redis, memcached, or similar). We would like to avoid this if possible, to avoid maintaining more services. 2) Improve SimpleTcpCluster to resend lost sessions. We had a brief look at the code and realize it's a bit of work. Before we dig deeper into this, it'd be good to see if someone has any thoughts on this, other suggestions or ideas to remedy the situation. 1. Use the BackupManager rather than the DeltaManager. It also scales better. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org -- *Linus Brimstedt * CTO +46 70 - 683 98 54 linus.brimst...@viskan.se *Viskan Distanshandel System AB* Druveforsvägen 8 504 33 Borås, Sweden +46 33 - 20 60 20 www.viskan.se
Re: File descriptors peaks with latest stable build of Tomcat 7
Am 22.04.2015 um 00:08 schrieb André Warnier: ... The OP has a complex setup, where we are not even sure that the various connections in various states are even related directly to Tomcat or not. Graphically, we have this : client -- TCP -- nginx -- TCP -- Tomcat -- webapp -- TCP -- external servers The output of netstat shows all the connections and their state, at the OS level. Even assuming that nginx runs on a separate host, that still leaves the possibility that most of the connections in CLOSE_WAIT state for example, would be connections between the webapps and external servers, having not much to do with Tomcat per se. But of course they use fd's and resources, just like the others. And for lsof, they would appear as belonging to the Tomcat process. See my response from 1.5 days ago which contains the individual statistics for each of the above three TCP parts. Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Rainer Jung wrote: Am 22.04.2015 um 00:08 schrieb André Warnier: ... The OP has a complex setup, where we are not even sure that the various connections in various states are even related directly to Tomcat or not. Graphically, we have this : client -- TCP -- nginx -- TCP -- Tomcat -- webapp -- TCP -- external servers The output of netstat shows all the connections and their state, at the OS level. Even assuming that nginx runs on a separate host, that still leaves the possibility that most of the connections in CLOSE_WAIT state for example, would be connections between the webapps and external servers, having not much to do with Tomcat per se. But of course they use fd's and resources, just like the others. And for lsof, they would appear as belonging to the Tomcat process. See my response from 1.5 days ago which contains the individual statistics for each of the above three TCP parts. Yes, sorry Rainer, I did not read that as carefully as I should have. And I do agree that the two main things which the OP should have a good look at, are - the nginx settings for keep-alive (client-nginx side) - the various webapp-to-external servers connections in CLOSE_WAIT state Collectively, these things must be using a lot of resources on the server, and probably slow it down significantly. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
What concerns me the most is the CLOSE_WAIT on tomcat side because when an fd peak appears the web application appears to be stuck. It feels like all its connections are consumed and none can be established from nginx anymore. Shouldn't the CLOSE_WAIT connection be recycled to received new connections from nginx ? I am less concerned by the webapp-to-external servers connections in CLOSE_WAIT state, these connections are handled using httpAsyncClient with a pool system that has a defined size and an eviction strategy (if too many connections are opened new connections request will only wait for say 100 ms before failling). We will look into the configuration you advice to check on nginx and tomcat size to see how it's setup. Thanks Thomas 2015-04-22 11:38 GMT+02:00 André Warnier a...@ice-sa.com: Rainer Jung wrote: Am 22.04.2015 um 00:08 schrieb André Warnier: ... The OP has a complex setup, where we are not even sure that the various connections in various states are even related directly to Tomcat or not. Graphically, we have this : client -- TCP -- nginx -- TCP -- Tomcat -- webapp -- TCP -- external servers The output of netstat shows all the connections and their state, at the OS level. Even assuming that nginx runs on a separate host, that still leaves the possibility that most of the connections in CLOSE_WAIT state for example, would be connections between the webapps and external servers, having not much to do with Tomcat per se. But of course they use fd's and resources, just like the others. And for lsof, they would appear as belonging to the Tomcat process. See my response from 1.5 days ago which contains the individual statistics for each of the above three TCP parts. Yes, sorry Rainer, I did not read that as carefully as I should have. And I do agree that the two main things which the OP should have a good look at, are - the nginx settings for keep-alive (client-nginx side) - the various webapp-to-external servers connections in CLOSE_WAIT state Collectively, these things must be using a lot of resources on the server, and probably slow it down significantly. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Getting Apache 2.2.29 to work with tomcat 8.0.21 using mod_jk 1.2.40 on a mac
Hey Mayuresh, On Wed, Apr 22, 2015 at 5:33 AM, Mayuresh mayuresh.kshirsa...@gmail.com wrote: Hi Gurus, i am trying to make this combination and am stumbled at a place. After a lot of searching and experimenting I finally decided to post this question here. Welcome to the mailing list! I am trying to load a page which loads just fine directly on tomcat. However if I try to load it through the Apache web server, it fails with a 502 on apache (400 on tomcat) For an unsuccessful request through apache I see the following in the logs, where all the headers seem to be garbled: 22-Apr-2015 01:53:46.656 FINE [http-nio-8080-exec-1] Right there - you are using HTTP connector to serve this request, as indicated by the thread name (http-nio-bio-exec-1). What is your relevant Apache (httpd) configuration for forwarding to Tomcat, e.g. your worker file that defines how the traffic is routed to tomcat instance, the mount point that defines how certain url is registered with different workers, etc... Typically you would connect to port 8009 (AJP) if you are connecting through MOD_JK Apache (httpd) module, i.e. worker..type=ajp worker..host=localhost worker..port=8009 (my suspicion is that you have 8080 defined here) See more information on workers file here: https://tomcat.apache.org/connectors-doc/reference/workers.html org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@616702aa Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:53:46.657 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [^R4^B^G^B^B^@^HHTTP/1.1^@^@^M/greeting/abc^@^@ 10.0.0.23^@ÿÿ^@^Posxltmkshi.local^@ü^@^@ ^K^@^Uosxltmkshi.local:9980^@ ^F^@ keep-alive^@ ^L^@^Hno-cache^@^@^MCache-Control^@^@^Hno-cache^@ ^A^@Jtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^@ ^N^@wMozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^@^@^OAccept-Encoding^@^@^Sgzip, deflate, sdch^@^@^OAccept-Language^@^@^Nen-US,en;q=0.8^@^@^OX-Forwarded-For^@^@^L14.102.224.6^@ ^H^@^A0^@ ^@^OAJP_REMOTE_PORT^@^@^E55806^@ ^@^PJK_LB_ACTIVATION^@^@^CACT^@ÿ] 0:0:0:0:0:0:0:1 - - [22/Apr/2015:01:47:03 -0700] - 400 - For Successful page load directly through tomcat I see the following in the log: 22-Apr-2015 01:48:56.176 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.createManagedBean Setting name: org.apache.coyote.RequestInfo 22-Apr-2015 01:48:56.176 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@630018d0 Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:48:56.178 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [GET /greeting/abcde HTTP/1.1^M Host: localhost:8080^M Connection: keep-alive^M Cache-Control: max-age=0^M Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^M User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^M Accept-Encoding: gzip, deflate, sdch^M Accept-Language: en-US,en;q=0.8^M Cookie: JSESSIONID=CB0590CD16933826596C4E78AE4FDB5F^M X-Forwarded-For: 14.102.224.6^M ^M ] 127.0.0.1 - - [22/Apr/2015:01:48:56 -0700] GET /greeting/abcde HTTP/1.1 200 1726 127.0.0.1 - - [22/Apr/2015:01:49:00 -0700] GET /greetingab.js HTTP/1.1 200 4 I can't seem to find why this is happening. Can you please help me figure out what the problem might be? So, let us know how are you connecting your Apache (httpd) to Tomcat instance. That's a good starting point for investigation. Cheers! Neven
Re: File descriptors peaks with latest stable build of Tomcat 7
Am 22.04.2015 um 11:38 schrieb André Warnier: Rainer Jung wrote: See my response from 1.5 days ago which contains the individual statistics for each of the above three TCP parts. Yes, sorry Rainer, I did not read that as carefully as I should have. No worries at all. Lots of stuff going back and forth. Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: File descriptors peaks with latest stable build of Tomcat 7
Am 22.04.2015 um 11:58 schrieb Thomas Boniface: What concerns me the most is the CLOSE_WAIT on tomcat side because when an fd peak appears the web application appears to be stuck. It feels like all its connections are consumed and none can be established from nginx anymore. Shouldn't the CLOSE_WAIT connection be recycled to received new connections from nginx ? Just to clarify: Every connection has two ends. In netstat the local end is left, the remote end is right. If a connection is between processes both on the same system, it will be shown in netstat twice. Once for each endpoint being the local side. CLOSE_WAIT for a connection between a (local) and b (remote) means, that b has closed the connection but not a. There is no automatism for a closing it because b has closed it. If CLOSE_WAIT pile up, then the idea of b and a when a connection should no longer be used are disparate. E.g. they might have very different idle timeouts (Keep Alive Timeout on HTTP speak), or one observed a problem that the other didn't observe. When I did the counting for Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT the 127.0.0.1:8080 was left in netstat output, so local. It means the other side (whatever is the other side of the connection, likely nginx) has closed the connection alardy, but not Tomcat. And the total number of those connections: Count IP:Port ConnectionState 8381127.0.0.1:8080 CLOSE_WAIT 1650127.0.0.1:8080 ESTABLISHED indeed sums up to the default maxConnections 1 mentioned by Chris. What I do not understand is, that the same connections looked at from nginx being the local end, show a totally different statistics: Count IP:Port ConnectionState 20119127.0.0.1:8080 SYN_SENT 4692127.0.0.1:8080 ESTABLISHED 488127.0.0.1:8080 FIN_WAIT2 122127.0.0.1:8080 TIME_WAIT 13127.0.0.1:8080 FIN_WAIT1 But maybe that's a problem to solve after you fixed the CLOSED_WAIT (or the 1000 limit) and redo the whole observation. Pretty big numbers you habe ... Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Getting Apache 2.2.29 to work with tomcat 8.0.21 using mod_jk 1.2.40 on a mac
Am 22.04.2015 um 11:33 schrieb Mayuresh: Hi Gurus, i am trying to make this combination and am stumbled at a place. After a lot of searching and experimenting I finally decided to post this question here. I am trying to load a page which loads just fine directly on tomcat. However if I try to load it through the Apache web server, it fails with a 502 on apache (400 on tomcat) For an unsuccessful request through apache I see the following in the logs, where all the headers seem to be garbled: 22-Apr-2015 01:53:46.656 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@616702aa Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:53:46.657 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [^R4^B^G^B^B^@^HHTTP/1.1^@^@^M/greeting/abc^@^@ 10.0.0.23^@ÿÿ^@^Posxltmkshi.local^@ü^@^@ ^K^@^Uosxltmkshi.local:9980^@ ^F^@ keep-alive^@ ^L^@^Hno-cache^@^@^MCache-Control^@^@^Hno-cache^@ ^A^@Jtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^@ ^N^@wMozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^@^@^OAccept-Encoding^@^@^Sgzip, deflate, sdch^@^@^OAccept-Language^@^@^Nen-US,en;q=0.8^@^@^OX-Forwarded-For^@^@^L14.102.224.6^@ ^H^@^A0^@ ^@^OAJP_REMOTE_PORT^@^@^E55806^@ ^@^PJK_LB_ACTIVATION^@^@^CACT^@ÿ] mod_jk talks the AJP protocol, so you need to connect it to the Tomcat AJP connector. You instead connected it to the Tomcat HTTP connector. You probably have configured port 8080 in the mod_jk config instead of 8009 (default AJP port in Tomcat default server.xml). Regards, Rainer - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Getting Apache 2.2.29 to work with tomcat 8.0.21 using mod_jk 1.2.40 on a mac
Hi Gurus, i am trying to make this combination and am stumbled at a place. After a lot of searching and experimenting I finally decided to post this question here. I am trying to load a page which loads just fine directly on tomcat. However if I try to load it through the Apache web server, it fails with a 502 on apache (400 on tomcat) For an unsuccessful request through apache I see the following in the logs, where all the headers seem to be garbled: 22-Apr-2015 01:53:46.656 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@616702aa Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:53:46.657 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [^R4^B^G^B^B^@^HHTTP/1.1^@^@^M/greeting/abc^@^@ 10.0.0.23^@ÿÿ^@^Posxltmkshi.local^@ü^@^@ ^K^@^Uosxltmkshi.local:9980^@ ^F^@ keep-alive^@ ^L^@^Hno-cache^@^@^MCache-Control^@^@^Hno-cache^@ ^A^@Jtext/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^@ ^N^@wMozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^@^@^OAccept-Encoding^@^@^Sgzip, deflate, sdch^@^@^OAccept-Language^@^@^Nen-US,en;q=0.8^@^@^OX-Forwarded-For^@^@^L14.102.224.6^@ ^H^@^A0^@ ^@^OAJP_REMOTE_PORT^@^@^E55806^@ ^@^PJK_LB_ACTIVATION^@^@^CACT^@ÿ] 0:0:0:0:0:0:0:1 - - [22/Apr/2015:01:47:03 -0700] - 400 - For Successful page load directly through tomcat I see the following in the log: 22-Apr-2015 01:48:56.176 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.modules.MbeansDescriptorsIntrospectionSource.createManagedBean Setting name: org.apache.coyote.RequestInfo 22-Apr-2015 01:48:56.176 FINE [http-nio-8080-exec-1] org.apache.tomcat.util.modeler.BaseModelMBean.preRegister preRegister org.apache.coyote.RequestInfo@630018d0 Catalina:type=RequestProcessor,worker=http-nio-8080,name=HttpRequest1 22-Apr-2015 01:48:56.178 FINE [http-nio-8080-exec-1] org.apache.coyote.http11.AbstractNioInputBuffer.parseRequestLine Received [GET /greeting/abcde HTTP/1.1^M Host: localhost:8080^M Connection: keep-alive^M Cache-Control: max-age=0^M Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8^M User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/42.0.2311.90 Safari/537.36^M Accept-Encoding: gzip, deflate, sdch^M Accept-Language: en-US,en;q=0.8^M Cookie: JSESSIONID=CB0590CD16933826596C4E78AE4FDB5F^M X-Forwarded-For: 14.102.224.6^M ^M ] 127.0.0.1 - - [22/Apr/2015:01:48:56 -0700] GET /greeting/abcde HTTP/1.1 200 1726 127.0.0.1 - - [22/Apr/2015:01:49:00 -0700] GET /greetingab.js HTTP/1.1 200 4 I can't seem to find why this is happening. Can you please help me figure out what the problem might be? Thanks, Mayuresh
