Re: Expected behavior of Tomcat under load
Hi Peter / Chris My sincere apologies, as I somehow missed this original reply from Peter :( Operating system and version? Java version? ubuntu@ip-10-202-99-31:~/configs$ uname -a Linux ip-10-202-99-31 2.6.35-24-virtual #42-Ubuntu SMP Thu Dec 2 05:15:26 UTC 2010 x86_64 GNU/Linux ubuntu@ip-10-202-99-31:~/configs$ java -version java version 1.6.0_25 Java(TM) SE Runtime Environment (build 1.6.0_25-b06) Java HotSpot(TM) 64-Bit Server VM (build 20.0-b11, mixed mode) Tomcat version: 6.0.32 - thanks for this. Java or native connector? I believe Java - its the default Zip installation extracted. With just the thread pool enabled on 8080 connector, with 300 threads. Pure Tomcat or something else in front? (I'm assuming pure Tomcat from what else you say) Nothing in front I guess the tcodump holds the key to seeing this happen. Note that this all hapens within just one second. The load was through an ESB that uses NIO - and I believe this really pushes data on all sockets at the same time. No. TimeSourceDestination Protocol Src Port Dst Port Info 389961 37.056567 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [SYN] Seq=0 Win=5792 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 TSV=363753 TSER=363574 WS=7 391297 37.108766 10.101.29.42 10.77.69.8TCP 8080 9062 8080 9062 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=363383 TSER=363753 WS=7 391298 37.108773 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [ACK] Seq=1 Ack=1 Win=5888 [TCP CHECKSUM INCORRECT] Len=0 TSV=363758 TSER=363383 391893 37.115809 10.77.69.810.101.29.42 HTTP 9062 8080 POST /xfire/xfire-service HTTP/1.1 391894 37.115837 10.77.69.810.101.29.42 HTTP 9062 8080 Continuation or non-HTTP traffic[Packet size limited during capture] 392677 37.125492 10.101.29.42 10.77.69.8TCP 8080 9062 8080 9062 [RST] Seq=1 Win=0 Len=0 I'll be happy to share any more information as required. thanks asankha
Re: Expected behavior of Tomcat under load
Something puzzles me since your first post : Asankha Perera wrote: ... No. TimeSourceDestination Protocol Src Port Dst Port Info 389961 37.056567 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [SYN] Seq=0 Win=5792 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 TSV=363753 TSER=363574 WS=7 ... What is this TCP CHECKSUM INCORRECT thing ? This is the output of some protocol analyser thing, right ? So it is totally independent of Tomcat or whatever. This packet is one that comes from whatever your client is, toward Tomcat. Why does it show that message ? And if that message can be believed, is it then not normal that the protocol stack which receives that (bad) TCP packet would reject it, and break the connection ? - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Expected behavior of Tomcat under load
From: André Warnier [mailto:a...@ice-sa.com] Subject: Re: Expected behavior of Tomcat under load What is this TCP CHECKSUM INCORRECT thing ? Usually because the checksum for outbound messages is being computed by the NIC rather than the TCP/IP stack software, so the protocol analyzer doesn't see the actual value. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of Tomcat under load
Something puzzles me since your first post : ... What is this TCP CHECKSUM INCORRECT thing ? This is the output of some protocol analyser thing, right ? Yes, its a capture from tcpdump, analyzed by wireshark So it is totally independent of Tomcat or whatever. This packet is one that comes from whatever your client is, toward Tomcat. Why does it show that message ? And if that message can be believed, is it then not normal that the protocol stack which receives that (bad) TCP packet would reject it, and break the connection ? I guess this is normal. I did a quick search and came across the following: http://www.ethereal.com/lists/ethereal-dev/200406/msg00090.html http://stackoverflow.com/questions/667848/java-socket-tcp-checksum-incorrect http://wiki.wireshark.org/TCP_Checksum_Verification This trace is from a EC2 node ubuntu@ip-10-202-99-31:~/configs$ ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off thanks asankha -- Asankha C. Perera AdroitLogic, http://adroitlogic.org http://esbmagic.blogspot.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
RE: Expected behavior of Tomcat under load
From: Asankha Perera [mailto:asankha.apa...@gmail.com] On Behalf Of Asankha C. Perera Subject: Re: Expected behavior of Tomcat under load Offload parameters for eth0: rx-checksumming: on tx-checksumming: on As expected, the NIC is handling the checksumming. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of Tomcat under load
Putting your answer together with the one from Chuck : I understand that if the tcpdump program runs on the same host as the one which is sending the packets, it may not be able to correctly see the TCP checksum, since it captures the packet before it goes out on the network, and it is the NIC which calculates and inserts the TCP checksum just before the packet is sent over the network. Right ? But is this the case here ? Where is/was the tcpdump program run, which captured these packets, as compared to the client and server systems ? Asankha C. Perera wrote: Something puzzles me since your first post : ... What is this TCP CHECKSUM INCORRECT thing ? This is the output of some protocol analyser thing, right ? Yes, its a capture from tcpdump, analyzed by wireshark So it is totally independent of Tomcat or whatever. This packet is one that comes from whatever your client is, toward Tomcat. Why does it show that message ? And if that message can be believed, is it then not normal that the protocol stack which receives that (bad) TCP packet would reject it, and break the connection ? I guess this is normal. I did a quick search and came across the following: http://www.ethereal.com/lists/ethereal-dev/200406/msg00090.html http://stackoverflow.com/questions/667848/java-socket-tcp-checksum-incorrect http://wiki.wireshark.org/TCP_Checksum_Verification This trace is from a EC2 node ubuntu@ip-10-202-99-31:~/configs$ ethtool -k eth0 Offload parameters for eth0: rx-checksumming: on tx-checksumming: on scatter-gather: on tcp-segmentation-offload: on udp-fragmentation-offload: off generic-segmentation-offload: off generic-receive-offload: off large-receive-offload: off ntuple-filters: off receive-hashing: off thanks asankha - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of Tomcat under load
On 05/26/2011 09:50 PM, André Warnier wrote: Putting your answer together with the one from Chuck : I understand that if the tcpdump program runs on the same host as the one which is sending the packets, it may not be able to correctly see the TCP checksum, since it captures the packet before it goes out on the network, and it is the NIC which calculates and inserts the TCP checksum just before the packet is sent over the network. Right ? But is this the case here ? Where is/was the tcpdump program run, which captured these packets, as compared to the client and server systems ? I am quite certain this was from the ESB node which was the client to tomcat .. thanks asankha -- Asankha C. Perera AdroitLogic, http://adroitlogic.org http://esbmagic.blogspot.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of Tomcat under load
Asankha C. Perera wrote: On 05/26/2011 09:50 PM, André Warnier wrote: Putting your answer together with the one from Chuck : I understand that if the tcpdump program runs on the same host as the one which is sending the packets, it may not be able to correctly see the TCP checksum, since it captures the packet before it goes out on the network, and it is the NIC which calculates and inserts the TCP checksum just before the packet is sent over the network. Right ? But is this the case here ? Where is/was the tcpdump program run, which captured these packets, as compared to the client and server systems ? I am quite certain this was from the ESB node which was the client to tomcat .. In that case it looks indeed like a false alarm. In the second link which you provided, I note this : quote As this may be confusing and will prevent Wireshark from reassemble TCP segments it's a good idea to switch checksum verification off in these cases. To disable checking of the TCP checksum validity, go to the TCP preferences and untick the box for checksum verification unquote Doing this may thus allow wireshark to better reassemble the packets and make the display somewhat clearer. Maybe avoiding these in the process : 391894 37.115837 10.77.69.810.101.29.42 HTTP 9062 8080 Continuation or non-HTTP traffic[Packet size limited during capture] and see more clearly what is being POST-ed. (Maybe it contains a clue about the reset ?) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Expected behavior of Tomcat under load
Hi All During some performance tests, we've seen that Tomcat resets TCP connections under high load. To reproduce this rather consistently, a thread pool with a maximum of 300 threads could be configured on default Tomcat 6.0.32, and then 1280 ~ 2560 concurrent user requests simulated from a different machine over a real NW interface. I assume this could be reproduced with proportionately smaller numbers for both as well. The implementation uses a Xfire soap service. Tomcat refusing connections, or taking longer to accept new connections, or taking longer to reply (causing a socket timeout) can be expected under such load - but what we see are TCP resets of connections to which a client has already sent a full HTTP request. Is this the default behavior of Tomcat? The problem this presents is that the client cannot safely fail over to another instance, unlike with a refused connection or a connect timeout (i.e. delay in accepting) thanks asankha No. TimeSourceDestination Protocol Src Port Dst Port Info 389961 37.056567 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [SYN] Seq=0 Win=5792 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 TSV=363753 TSER=363574 WS=7 391297 37.108766 10.101.29.42 10.77.69.8 TCP 8080 9062 8080 9062 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=363383 TSER=363753 WS=7 391298 37.108773 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [ACK] Seq=1 Ack=1 Win=5888 [TCP CHECKSUM INCORRECT] Len=0 TSV=363758 TSER=363383 391893 37.115809 10.77.69.810.101.29.42 HTTP 9062 8080 POST /xfire/xfire-service HTTP/1.1 391894 37.115837 10.77.69.810.101.29.42 HTTP 9062 8080 Continuation or non-HTTP traffic[Packet size limited during capture] 392677 37.125492 10.101.29.42 10.77.69.8 TCP 8080 9062 8080 9062 [RST] Seq=1 Win=0 Len=0 -- Asankha C. Perera AdroitLogic, http://adroitlogic.org http://esbmagic.blogspot.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Expected behavior of Tomcat under load
Operating system and version? Java version? Tomcat version: 6.0.32 - thanks for this. Java or native connector? Pure Tomcat or something else in front? (I'm assuming pure Tomcat from what else you say) - Peter On 25 May 2011 12:51, Asankha C. Perera asan...@apache.org wrote: Hi All During some performance tests, we've seen that Tomcat resets TCP connections under high load. To reproduce this rather consistently, a thread pool with a maximum of 300 threads could be configured on default Tomcat 6.0.32, and then 1280 ~ 2560 concurrent user requests simulated from a different machine over a real NW interface. I assume this could be reproduced with proportionately smaller numbers for both as well. The implementation uses a Xfire soap service. Tomcat refusing connections, or taking longer to accept new connections, or taking longer to reply (causing a socket timeout) can be expected under such load - but what we see are TCP resets of connections to which a client has already sent a full HTTP request. Is this the default behavior of Tomcat? The problem this presents is that the client cannot safely fail over to another instance, unlike with a refused connection or a connect timeout (i.e. delay in accepting) thanks asankha No. TimeSourceDestination Protocol Src Port Dst Port Info 389961 37.056567 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [SYN] Seq=0 Win=5792 [TCP CHECKSUM INCORRECT] Len=0 MSS=1460 TSV=363753 TSER=363574 WS=7 391297 37.108766 10.101.29.42 10.77.69.8TCP 8080 9062 8080 9062 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=363383 TSER=363753 WS=7 391298 37.108773 10.77.69.810.101.29.42 TCP 9062 8080 9062 8080 [ACK] Seq=1 Ack=1 Win=5888 [TCP CHECKSUM INCORRECT] Len=0 TSV=363758 TSER=363383 391893 37.115809 10.77.69.810.101.29.42 HTTP 9062 8080 POST /xfire/xfire-service HTTP/1.1 391894 37.115837 10.77.69.810.101.29.42 HTTP 9062 8080 Continuation or non-HTTP traffic[Packet size limited during capture] 392677 37.125492 10.101.29.42 10.77.69.8TCP 8080 9062 8080 9062 [RST] Seq=1 Win=0 Len=0 -- Asankha C. Perera AdroitLogic, http://adroitlogic.org http://esbmagic.blogspot.com - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
