I just noticed in my ethereal traffic that it is the load testing client that is initiating the tcp reset packets so this may be unrelated to mina as the client currently doesn't use mina.
The server is runing mina 0.7.3 using jdk1.5.0_04 and the os/kernel is redhat 2.4.21-20.ELsmp The client is fedora 2.6.10-1.741_FC3smp What is strange is that I don't see any tcp errors on either machine using netstat -i eth0 -t -c 10 Both jvms have ample memory allocated. After running the load test for about 10 seconds I start to see occasional read timeouts on the client. First I thought the mina server was not responding but then I found that the client is actually issuing a tcp reset before actually trying to send it's payload to the mina server. As a result the mina server never actually accepts the packet and the client after doing a tcp retransmission a few times eventually times out. Any suggestions? Here is a sample of what I'm seeing on the wire: client 10.1.36.45 mina server 206.246.214.14 No. Time Source Destination Protocol Info 36334 36.252292 10.1.36.45 206.246.213.14 TCP 41622 > 11111 [SYN] Seq=0 Ack=0 Win=5840 Len=0 MSS=1460 TSV=607287694 TSER=0 WS=2 36335 36.252323 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [SYN, ACK] Seq=0 Ack=1 Win=5792 Len=0 MSS=1460 TSV=262940360 TSER=607287694 WS=0 36336 36.252583 10.1.36.45 206.246.213.14 TCP 41622 > 11111 [RST] Seq=1 Ack=3719067569 Win=23168 Len=0 36337 36.252669 10.1.36.45 206.246.213.14 TCP 41622 > 11111 [ACK] Seq=1 Ack=1 Win=5840 Len=0 TSV=607287694 TSER=262940360 36338 36.252681 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 36341 36.253272 10.1.36.45 206.246.213.14 TCP 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607287695 TSER=262940360 36342 36.253286 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 36683 36.458444 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607287900 TSER=262940360 36684 36.458469 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 37496 36.868383 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607288310 TSER=262940360 37497 36.868405 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 39146 37.688160 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607289130 TSER=262940360 39147 37.688178 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 42427 39.328107 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607290770 TSER=262940360 42428 39.328171 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 47930 42.607621 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607294050 TSER=262940360 47931 42.607651 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 51244 44.795623 10.1.36.45 206.246.213.14 TCP 41622 > 11111 [FIN, ACK] Seq=132 Ack=1 Win=5840 Len=0 TSV=607296239 TSER=262940360 51245 44.795631 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 59141 49.165337 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607300610 TSER=262940360 59142 49.165362 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 80975 62.283799 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607313730 TSER=262940360 80976 62.283822 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 102661 88.518412 10.1.36.45 206.246.213.14 TCP [TCP Retransmission] 41622 > 11111 [FIN, PSH, ACK] Seq=1 Ack=1 Win=5840 Len=131 TSV=607339970 TSER=262940360 102662 88.518428 206.246.213.14 10.1.36.45 TCP 11111 > 41622 [RST] Seq=1 Ack=3289563097 Win=0 Len=0 Thanks, Alex. On Wed, 13 Jul 2005, Vinod Panicker wrote: > During load testing, I've also experienced connection resets and > drops, but they disappeared when I gave the JVM more memory to play > with. Search in the archives for this (-Xms and something else i > forget). > > Regards, > Vinod. > > On 7/13/05, Trustin Lee <[EMAIL PROTECTED]> wrote: > > Hi Alex, > > > > > > 2005/7/13, Alex Burmester <[EMAIL PROTECTED]>: > > > Hi Trustin, I have built a protocol router on top of mina > > > and it generally runs very well. > > > > > > Great. > > > > > I have found that when I fire up about 100 threads and pound on it with my > > > load testing client that I do get a few lost packets that never make it to > > > the mina server process. I did a tcpdump and found that the server > > > machine is issuing tcp resets on the packets that are not showing up. > > > Not sure why yet. I assume that it's some tcp parameters I need to bump > > > up on the linux box but I thought I would check with you to see if there > > > are any tunable parameters in the mina framework or in the jvm that you > > > know of that I should check. > > > > > > Well, are you using MINA 0.7.3? Please try to upgrade to 0.7.3 if you're > > using an older version. You can configure all socket parameters by calling > > IoSession.getConfig() and downcasting the returned object into > > SocketSessionConfig. To configure ServerSocketChannel, you'll have to > > upgrade to 0.9 for now. > > > > Please let me know the result after you upgrade to 0.7.3. > > > > Trustin-- > > what we call human nature is actually human habit > > -- > > http://gleamynode.net/ >
