Re: Fragmented delivery of servlet request
Thanks Chris for replying, 1) I have not looked at using mod_jk as a connector. Do you believe mod_jk is a better option to use than mod_proxy_ajp ? 2) proxy_ajp.conf just points to localhost on port 8009 /usr/sbin/httpd -v Server version: Apache/2.2.3 Server built: Mar 27 2010 13:52:45 Server is a vanilla Centos5.4 linux install. 3) I checked the documentation on mod_jk, http://tomcat.apache.org/connectors-doc/reference/workers.html and see a reply_timout worker directive, but cannot find a request_timeout directive. What is the theory behind a slow tomcat request from apache ? eg. Should apache buffer up the client request before sending across the connector, or is the connector responsible for handling a tomcat request across many fragmented request packets ? 4) I will look at reproducing using a hello world example on a TEST server with the same OS and then look at your suggestion of upgrading to a new version. Our preference though is to stay with vanilla CENTOS, if we can. Easier for future software maintenance. Doug. --- On Sat, 10/4/10, Christopher Schultz wrote: > From: Christopher Schultz > Subject: Re: Fragmented delivery of servlet request > To: "Tomcat Users List" > Received: Saturday, 10 April, 2010, 7:48 AM > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA1 > > Doug, > > On 4/9/2010 2:31 PM, Doug Herbert wrote: > > some timimgs : > > > > took 1500 milli seconds for all 19 packets to arrive > at that web server, from the client's browser connected > across a 'slow' cellular network. > > > > took 2 ( two ) milli seconds for tomcat to balk at the > incoming request and send back the RSP packet, after the > first REQ packet sent into tomcat. > > > > eg. in plain terms tomcat started responding before > all fragmented packets had even arrived at the web > server ! > > Can you post your mod_proxy_ajp configuration? 2ms is a > very short > amount of time for the AJP listener to reject a request > because it is > incomplete, but I suppose it's possible. > > I see you're running httpd 2.2.3, which is probably a > package-managed > version from CentOS. Is that really 2.2.3, or does it have > some > additional patches on top of it? I ask because > mod_proxy_ajp has > improved considerably since 2.2.3 (the current version is > 2.2.15). > > Would it be possible for you to test against a more recent > version, even > if you can't upgrade in production quite yet? If this is > not possible, > how about switching (at least temporarily) to mod_jk, > compiled yourself? > It's possible that something in the last (quite a) few > versions has been > identified and fixed. Is that's the case, there's no reason > to chase our > tails re-identifying a bug that's already been fixed. > > If you have gcc and apxs handy, compiling mod_jk is super > easy, as is > installation and configuration. > > - -chris - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fragmented delivery of servlet request
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Doug, On 4/9/2010 2:31 PM, Doug Herbert wrote: > some timimgs : > > took 1500 milli seconds for all 19 packets to arrive at that web server, from > the client's browser connected across a 'slow' cellular network. > > took 2 ( two ) milli seconds for tomcat to balk at the incoming request and > send back the RSP packet, after the first REQ packet sent into tomcat. > > eg. in plain terms tomcat started responding before all fragmented packets > had even arrived at the web server ! Can you post your mod_proxy_ajp configuration? 2ms is a very short amount of time for the AJP listener to reject a request because it is incomplete, but I suppose it's possible. I see you're running httpd 2.2.3, which is probably a package-managed version from CentOS. Is that really 2.2.3, or does it have some additional patches on top of it? I ask because mod_proxy_ajp has improved considerably since 2.2.3 (the current version is 2.2.15). Would it be possible for you to test against a more recent version, even if you can't upgrade in production quite yet? If this is not possible, how about switching (at least temporarily) to mod_jk, compiled yourself? It's possible that something in the last (quite a) few versions has been identified and fixed. Is that's the case, there's no reason to chase our tails re-identifying a bug that's already been fixed. If you have gcc and apxs handy, compiling mod_jk is super easy, as is installation and configuration. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAku/hJ4ACgkQ9CaO5/Lv0PBbsQCghjkyTMumm8eQkJ63YS5sHYUo dMoAn3alHa51nNh0DvtWCY6PrMPavLpB =ZFu9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Fragmented delivery of servlet request
some timimgs : took 1500 milli seconds for all 19 packets to arrive at that web server, from the client's browser connected across a 'slow' cellular network. took 2 ( two ) milli seconds for tomcat to balk at the incoming request and send back the RSP packet, after the first REQ packet sent into tomcat. eg. in plain terms tomcat started responding before all fragmented packets had even arrived at the web server ! --- On Fri, 9/4/10, Doug Herbert wrote: > From: Doug Herbert > Subject: Fragmented delivery of servlet request > To: users@tomcat.apache.org > Received: Friday, 9 April, 2010, 3:09 PM > Thoughts welcomed on the following > problem : > > Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2, > java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5 > > > A large servlet request, POST'ed from client to apache, > connected using proxy_ajp to ajp://localhost:8009. > > reassembled on server eth0 as 10766 bytes but only > partially sent to tomcat. Th eth0 line trace to http ( port > 80 ) was split over 19 packets of 536 bytes each. > > My initial thoughts were, that maxHttpHeaderSize=8192, was > too low. Increasing to 16384 did not resolve the issue. > > So more wireshark line traces, ( one tcpdump across eth0 > capturing the http POST and a 2nd tcpdump across loopback > capturing ajp connector traffic ) revealed, that apache via > connector ajp delivered each packet realtime time to tomcat, > without waiting for all 10766 bytes to arrive, though the > trace across loopback on port 8009, revealed that tomcat > starting the reply before all 10766 bytes had arrived. > > tcpdump on eth0 confirmed, by reassembled tcp segment to > contained the 10766 bytes from the browser client. > > ( Note : I have mangled URI SRV referer host headers ) > > Apache JServ Protocol v1.3 > Magic: 1234 > Length: 528 > Code: (2) FORWARD REQUEST > Method: (4) POST > Version: HTTP/1.1 > URI: //y > RADDR: 192.168.252.68 > RHOST: > SRV: xxx > PORT: 80 > SSLP: 0 > NHDR: 11 > accept: image/gif, image/x-xbitmap, > image/jpeg, image/pjpeg, application/x-shockwave-flash, > application/vnd.ms-powerpoint, application/vnd.ms-excel, > application/msword, */* > referer: http:// > accept-language: en-us > content-type: > application/x-www-form-urlencoded > UA-CPU: x86 > accept-encoding: gzip, deflate > user-agent: Mozilla/4.0 (compatible; MSIE > 7.0; Windows NT 5.1) > host: xxx > content-length: 10766 > connection: Keep-Alive > Cache-Control: no-cache > > > My question is, why did tomcat start to send the RSP ( > response ) after only receiving the first data packet from > ajp connector, around 500 bytes, when the above states the > content length is 10766 bytes. You can also see that > subsequent REQ ( request body ) data packets are still > transmitted to tomcat, though the response is already being > returned. > > Another option I thought of, was to buffer the whole 10766 > bytes up on the apache side, and then get the connector to > pass the request across as one 'big' packet. Is this > possible to configure ? If so, then maxHttpHeaderSize will > come into play, though at the moment only many small packets > are being sent across in the fragmented request. > > A stack dump in catalina.out can be seen below, where the > input filter is balking on the POSTed parameters, most > likely because only 550 bytes of the 10766, have turned up > for the input filter to process. > > Apr 9, 2010 10:51:55 AM > org.apache.catalina.connector.Request parseParameters > WARNING: Exception thrown whilst processing POSTed > parameters > java.io.IOException: Socket read failed > at > org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038) > at > org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159) > at > org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091) > at > org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130) > at > org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115) > at > org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233) > at > org.apache.coyote.Request.doRead(Request.java:419) > at > org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265) > at > org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403) > at > org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280) >
Fragmented delivery of servlet request
Thoughts welcomed on the following problem : Centos 5.4, http 2.2.3, tomcat5-5.5.23-0jpp.7.el5_3.2, java-1.6.0-openjdk-1.6.0.0-1.7.b09.el5 A large servlet request, POST'ed from client to apache, connected using proxy_ajp to ajp://localhost:8009. reassembled on server eth0 as 10766 bytes but only partially sent to tomcat. Th eth0 line trace to http ( port 80 ) was split over 19 packets of 536 bytes each. My initial thoughts were, that maxHttpHeaderSize=8192, was too low. Increasing to 16384 did not resolve the issue. So more wireshark line traces, ( one tcpdump across eth0 capturing the http POST and a 2nd tcpdump across loopback capturing ajp connector traffic ) revealed, that apache via connector ajp delivered each packet realtime time to tomcat, without waiting for all 10766 bytes to arrive, though the trace across loopback on port 8009, revealed that tomcat starting the reply before all 10766 bytes had arrived. tcpdump on eth0 confirmed, by reassembled tcp segment to contained the 10766 bytes from the browser client. ( Note : I have mangled URI SRV referer host headers ) Apache JServ Protocol v1.3 Magic: 1234 Length: 528 Code: (2) FORWARD REQUEST Method: (4) POST Version: HTTP/1.1 URI: //y RADDR: 192.168.252.68 RHOST: SRV: xxx PORT: 80 SSLP: 0 NHDR: 11 accept: image/gif, image/x-xbitmap, image/jpeg, image/pjpeg, application/x-shockwave-flash, application/vnd.ms-powerpoint, application/vnd.ms-excel, application/msword, */* referer: http:// accept-language: en-us content-type: application/x-www-form-urlencoded UA-CPU: x86 accept-encoding: gzip, deflate user-agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1) host: xxx content-length: 10766 connection: Keep-Alive Cache-Control: no-cache My question is, why did tomcat start to send the RSP ( response ) after only receiving the first data packet from ajp connector, around 500 bytes, when the above states the content length is 10766 bytes. You can also see that subsequent REQ ( request body ) data packets are still transmitted to tomcat, though the response is already being returned. Another option I thought of, was to buffer the whole 10766 bytes up on the apache side, and then get the connector to pass the request across as one 'big' packet. Is this possible to configure ? If so, then maxHttpHeaderSize will come into play, though at the moment only many small packets are being sent across in the fragmented request. A stack dump in catalina.out can be seen below, where the input filter is balking on the POSTed parameters, most likely because only 550 bytes of the 10766, have turned up for the input filter to process. Apr 9, 2010 10:51:55 AM org.apache.catalina.connector.Request parseParameters WARNING: Exception thrown whilst processing POSTed parameters java.io.IOException: Socket read failed at org.apache.coyote.ajp.AjpAprProcessor.read(AjpAprProcessor.java:1038) at org.apache.coyote.ajp.AjpAprProcessor.readMessage(AjpAprProcessor.java:1159) at org.apache.coyote.ajp.AjpAprProcessor.receive(AjpAprProcessor.java:1091) at org.apache.coyote.ajp.AjpAprProcessor.refillReadBuffer(AjpAprProcessor.java:1130) at org.apache.coyote.ajp.AjpAprProcessor.access$0(AjpAprProcessor.java:1115) at org.apache.coyote.ajp.AjpAprProcessor$SocketInputBuffer.doRead(AjpAprProcessor.java:1233) at org.apache.coyote.Request.doRead(Request.java:419) at org.apache.catalina.connector.InputBuffer.realReadBytes(InputBuffer.java:265) at org.apache.tomcat.util.buf.ByteChunk.substract(ByteChunk.java:403) at org.apache.catalina.connector.InputBuffer.read(InputBuffer.java:280) at org.apache.catalina.connector.CoyoteInputStream.read(CoyoteInputStream.java:193) at org.apache.catalina.connector.Request.readPostBody(Request.java:2419) at org.apache.catalina.connector.Request.parseParameters(Request.java:2398) at org.apache.catalina.connector.Request.getParameter(Request.java:1005) at org.apache.catalina.connector.RequestFacade.getParameter(RequestFacade.java:353) at com..y.filters.RequestValidation.getParameter(RequestValidation.java:40) at com...z.service(z.java:157) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at com...filters.RequestFilter.doFilter(RequestFilter.java:16) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.Standa
