[c-nsp] Network performance question - TCP Window issue?
Hi Guys, (Apologies if this is too off-topic for the cisco list) Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's Latency: A - B, 2 - 3 m/sec (No packet loss, minimal jitter) B - C, 12 - 13 m/sec (No packet loss, minimal jitter) Performing wget or iperf I see the following (Have linux servers at each POP) Pulling file from B-A (A's linux server is on 10/100 port) I get 9.59M/s (So basically 100Mbit) Pulling file from C-B I get 8.30M/s, but it does fluctuate up+down considerably Pulling file from C-A I get a flatline at 2.16M/s (Multiple sessions and all attain this speed, but no faster and doesnt fluctuate) Im not seeing any excessive errors/drops nor any duplex issues on any of the switches - Is TCP Window size the cause of the huge speed difference I see from C-B vs C-A, even though the latency difference between the 2 is minimal?(i.e. 3m/sec?) or does the extra hop cause the issue? thanks in advance for any advice. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
On 29/04/2012, at 5:34 PM, CiscoNSP_list CiscoNSP_list wrote: Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's Latency: A - B, 2 - 3 m/sec (No packet loss, minimal jitter) B - C, 12 - 13 m/sec (No packet loss, minimal jitter) There is a huge latency difference between A-B and B-C. (about 5 times!) How far are these sites apart? This might also make some good reading: http://knol.google.com/k/understanding-network-and-internet-latency Cheers Andrew ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
On Sun, 29 Apr 2012 18:04 +1030, CiscoNSP_list CiscoNSP_list wrote: Performing wget or iperf I see the following (Have linux servers at each POP) Pulling file from B-A (A's linux server is on 10/100 port) I get 9.59M/s (So basically 100Mbit) Pulling file from C-B I get 8.30M/s, but it does fluctuate up+down considerably Pulling file from C-A I get a flatline at 2.16M/s (Multiple sessions and all attain this speed, but no faster and doesnt fluctuate) Im not seeing any excessive errors/drops nor any duplex issues on any of the switches Did you run your iperf tests also with UDP? (The numbers don't look like it.) With TCP you won't see many drops on your switches, it will adjust - and you will see less throughput. With iperf available at all three sites I would run tests with UDP streams. This won't find the maximum bandwith automatically, you have to set a bandwidth for testing and see if you have any packet loss. Keep in mind that your carrier might police on ethernet bandwidth, iperf measures IP throuput. Best regards, Klaus Kastens -- Klaus Kastens NetUSE AG Dr.-Hell-Str. 6, D-24107 Kiel,Germany Fon: +49 431 2390 400 (07:00 UTC - 17:00 UTC) Fax: +49 431 2390 499 Vorstand: Dr. Joerg Posewang (Vorsitz), Dr. Roland Kaltefleiter, Andreas Seeger Aufsichtsrat: Dr. Dirk Lukas (Vorsitz) Sitz der AG: Kiel, HRB 5358 USt.ID: DE156073942 Diese E-Mail enthaelt vertrauliche oder rechtlich geschuetzte Informationen. Das unbefugte Kopieren dieser E-Mail oder die unbefugte Weitergabe der enthaltenen Informationen ist nicht gestattet. The information contained in this message is confidential or protected by law. Any unauthorised copying of this message or unauthorised distribution of the information contained herein is prohibited. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
On 29/04/2012, at 5:34 PM, CiscoNSP_list CiscoNSP_list wrote: Have 3 POPs A,B+C (A connected to B via 200Mb Eth, B connected to C via 250Mb Eth), all POPs are 7200's with 2960's and 3560's Latency: A - B, 2 - 3 m/sec (No packet loss, minimal jitter) B - C, 12 - 13 m/sec (No packet loss, minimal jitter) There is a huge latency difference between A-B and B-C. (about 5 times!) How far are these sites apart? A+B about 70K's B+C about 1000K's I've just checked in the opposite direction (C - A) and I see ~4.5M/s - A's on a 10/100 port, C's on a Gig port...so pulling from C-A would be Gig, 250, 200, 100 (where I see 2.15M/s), but in the other direction (pulling from A-C) would be 100, 200, 250, Gig... This might also make some good reading: http://knol.google.com/k/understanding-network-and-internet-latency Thanks - Will have a read. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
Did you run your iperf tests also with UDP? (The numbers don't look like it.) With TCP you won't see many drops on your switches, it will adjust - and you will see less throughput. With iperf available at all three sites I would run tests with UDP streams. This won't find the maximum bandwith automatically, you have to set a bandwidth for testing and see if you have any packet loss. Keep in mind that your carrier might police on ethernet bandwidth, iperf measures IP throuput. Thanks Klaus - No, did not test with udp...here it is: (With 100M had too many drops - 80M was the best:)[ 3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec[ 3] Sent 68029 datagrams[ 3] Server Report:[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec 0.044 ms1/68028 (0.0015%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order So to be able to see similar performance with tcp, I will need to adjust tcp window correct? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
I have seen this before, it's called bandwidth delay product and is linked to window size, let us know the tcp results after you have adjusted. Sent from my iPad On 29 Apr 2012, at 10:22, CiscoNSP_list CiscoNSP_list cisconsp_l...@hotmail.com wrote: Did you run your iperf tests also with UDP? (The numbers don't look like it.) With TCP you won't see many drops on your switches, it will adjust - and you will see less throughput. With iperf available at all three sites I would run tests with UDP streams. This won't find the maximum bandwith automatically, you have to set a bandwidth for testing and see if you have any packet loss. Keep in mind that your carrier might police on ethernet bandwidth, iperf measures IP throuput. Thanks Klaus - No, did not test with udp...here it is: (With 100M had too many drops - 80M was the best:)[ 3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec[ 3] Sent 68029 datagrams[ 3] Server Report:[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec 0.044 ms1/68028 (0.0015%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order So to be able to see similar performance with tcp, I will need to adjust tcp window correct? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
The timing of this is coincidental. I've been helping to troubleshoot a similar problem at work for days. Let's say we have three servers, A, B and C. We transfer files between them and here is what we see: A to B: Fast (around 18 MB/s) B to A: Slow (around 1 MB/s) A to C: Slow (around 1MB/s) C to A: Fast (around 18MB/s) In our case, Server A is fast when sending to B but not when sending to C. C can send at a high speed when sending back to A, though. We've checked everything we can think of. The paths aren't the same. One path goes through a firewall, another path goes through GRE tunnels. There are no TCP retransmits and we've verified that MTU isn't the problem. The firewall can't be the problem because it's only in the path of one set of transfers. All the TCP settings we've checked on the servers seem to be the same, although I'm not a server guy. Someone else has been checking those. The endpoints are on 1-gig links but it's 10-gig the whole way between them. There is about 50ms round-trip latency in all cases. I have no idea what could account for this behavior. On Sun, Apr 29, 2012 at 5:41 AM, sledge...@gmail.com wrote: I have seen this before, it's called bandwidth delay product and is linked to window size, let us know the tcp results after you have adjusted. Sent from my iPad On 29 Apr 2012, at 10:22, CiscoNSP_list CiscoNSP_list cisconsp_l...@hotmail.com wrote: Did you run your iperf tests also with UDP? (The numbers don't look like it.) With TCP you won't see many drops on your switches, it will adjust - and you will see less throughput. With iperf available at all three sites I would run tests with UDP streams. This won't find the maximum bandwith automatically, you have to set a bandwidth for testing and see if you have any packet loss. Keep in mind that your carrier might police on ethernet bandwidth, iperf measures IP throuput. Thanks Klaus - No, did not test with udp...here it is: (With 100M had too many drops - 80M was the best:)[ 3] local xxx.xxx.73.54 port 45790 connected with xxx.xxx.65.2 port 5001[ ID] Interval Transfer Bandwidth[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec[ 3] Sent 68029 datagrams[ 3] Server Report:[ 3] 0.0-10.0 sec 95.4 MBytes 80.0 Mbits/sec 0.044 ms 1/68028 (0.0015%)[ 3] 0.0-10.0 sec 1 datagrams received out-of-order So to be able to see similar performance with tcp, I will need to adjust tcp window correct? ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/ ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
On 30/04/2012, at 3:57 AM, John Neiberger wrote: The timing of this is coincidental. I've been helping to troubleshoot a similar problem at work for days. Let's say we have three servers, A, B and C. We transfer files between them and here is what we see: A to B: Fast (around 18 MB/s) B to A: Slow (around 1 MB/s) A to C: Slow (around 1MB/s) C to A: Fast (around 18MB/s) In our case, Server A is fast when sending to B but not when sending to C. C can send at a high speed when sending back to A, though. account for this behavior. How are you testing this speed? What servers with what connections with which operating system versions? What is exactly the list of all hardware/ interfaces/ connections between these 3 boxes. You need to draw this up. This however is probably a different issue to the original one listed. ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
Re: [c-nsp] Network performance question - TCP Window issue?
On 2012-04-29 19:57, John Neiberger wrote: The timing of this is coincidental. I've been helping to troubleshoot a similar problem at work for days. Let's say we have three servers, A, B and C. We transfer files between them and here is what we see: A to B: Fast (around 18 MB/s) B to A: Slow (around 1 MB/s) A to C: Slow (around 1MB/s) C to A: Fast (around 18MB/s) In our case, Server A is fast when sending to B but not when sending to C. C can send at a high speed when sending back to A, though. Typical problems with the different speed depending on the direction are caused either by duplex mismatch at access port (test, don't trust what one side tells you!) or problems with negotiating the TCP windows size (depending on the TCP/IP stack, application and tool you may get a number of different results). -- There's no sense in being precise when | Łukasz Bromirski you don't know what you're talking | jid:lbromir...@jabber.org about. John von Neumann |http://lukasz.bromirski.net ___ cisco-nsp mailing list cisco-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
