Paul, The bursts are sub-second, and you would not see them on the transfer rate over time or using 30-seconds average counters on the switches (you would see them with a sniffer...)
I suggest you test this in a structured way, so that we know where the problem is coming from, and then we know which part of the solution to fix. In order to test, you should repeat the download test with 100M and 1G settings at each of the following locations: 1. Same access switch 2. On the 6500 before the WAN connection 3. From Level 3 (beyond the WAN connection) The locations above are for the client ; the server should remain at the same place. Each location should be tested twice: With the server set to 100M and to 1G. Another thing is to validate that the WAN link is capable of providing a full 1G rate. If you can run an iperf test over it at 1G (outgoing!) it would allow us to validate this part. Thanks Arie -----Original Message----- From: Paul [mailto:[email protected]] Sent: Monday, June 28, 2010 18:58 To: Arie Vayner (avayner) Cc: Joe Loiacono; [email protected] Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over WANlinks This isn't exactly the problem I am seeing.. I actually set up a windows server and it shows the same result as the centos server which leads me to believe it's not a driver issue with centos. None of our connections are overflowing, the transfer doesn't even start out fast. It's going through all gigabit or higher ports the entire way. One particular transfer I get 1.3MB/s every time, consistently, and if i disable TSO/GSO i get 8-9MB/s average but during the transfer the rate jumps up and down a lot. (this is on centos using ethtool -K eth0 tso off gso off) Windows gets the 1.3MB/s but i haven't tried disabling tso/gso yet. If I set the port to 100mbits, both max it out no problem. Locally where latency is < 1ms both come near maxing out the gigabit port (probably hard drive limitation) Is there any utility that will test this end to end ? I've used iperf to do loss/transfer tests. What kills me is that the servers i'm using to test with can download at 300-400mbits from the server on the other end that im using to test with but can only upload at 10mbits. One would think, that if a server on level3 for example in one location and another server on level3 in another location both on gigabit ports, should get a good rate both directions. And of course if i set it to 100m, it gets 100m and not 10m.. I'm still stumped by this issue. Wouldn't having a server on the other end at 1gbit and using a 100m port to upload with cause more packet drops than having gigabit on the uploading server? since it maxes out the 100m port Arie Vayner (avayner) wrote: > Joe, this is exactly the phenomena I was referring to. It can be > controlled with applying shaping on platforms that can support this kind > of QOS policy (requires large buffers). > Usually available on WAN routers, specific switches or requires specific > modules on some other switches. > > Arie > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Joe Loiacono > Sent: Monday, June 28, 2010 17:11 > To: Paul > Cc: [email protected]; [email protected] > Subject: Re: [c-nsp] Centos upload speed slower on 1000m than 100m over > WANlinks > > OK I'm jumping in on this thread late as I just got back from some > vacation, don't know if this particluar observation has been discussed, > but ... > > We've seen this problem a lot when moving up to new local connection > speeds. The problem for us has been that unless the entire path can > support the new speed (e.g., 1G) switches down the path that connect to > slower speeds (e.g. 100M) will overflow and put your data transfer into > TCP slow-start recovery. As soon as the sending NIC is 'downgraded' > (e.g., > back to 100 M) the overflows disappear, slow-start is avoided, and > performance improves. Bitterly ironic. > > Joe > > > > From: > Paul <[email protected]> > To: > [email protected] > Date: > 06/27/2010 03:08 AM > Subject: > [c-nsp] Centos upload speed slower on 1000m than 100m over WAN links > > > > I'm not even sure this is the right forum but since we use mainly Cisco > equipment I'll give this a shot. :) > I have tried several centos based servers and compiled various kernels > and the results have been extremely weird. > 90% of the cases the remote hosts can download from a server at > 1-5megabytes per second, and most of these are over > the internet ranging from 30-200ms away. Local (1ms or less) is super > fast 100MB/s for example. > Ok that sounds normal since it's going over the internet, etc. But > here's the )(!...@*! part.. > If I set the port speed to 100 megabits full duplex on the switch and > server , the clients that get 1-5MB/s now get 11MB/s which is > approximately the limit of the 100mbit port. > Totally stumped here, tried different nics, servers, even 4 different > switches. Is a very interesting problem and I'm probing to see > if anyone else has encountered it. > So far the only OS i have tried is centos, but different versions and > kernels and hardware. > All the switches/routers are Cisco based, but I seriously doubt that has > > anything to do with this. :P > > -- GloboTech Communications Phone: 1-514-907-0050 x 215 Toll Free: 1-(888)-GTCOMM1 Fax: 1-(514)-907-0750 [email protected] http://www.gtcomm.net _______________________________________________ cisco-nsp mailing list [email protected] https://puck.nether.net/mailman/listinfo/cisco-nsp archive at http://puck.nether.net/pipermail/cisco-nsp/
