Re: increasing transmit speeds in WAN setting?
- Original Message - From: Moses Leslie [EMAIL PROTECTED] To: Ted Mittelstaedt [EMAIL PROTECTED] Cc: freebsd-questions@freebsd.org Sent: Friday, October 20, 2006 1:35 AM Subject: Re: increasing transmit speeds in WAN setting? On Thu, 19 Oct 2006, Ted Mittelstaedt wrote: Until you do what I told you to do and properly setup and test under fxp0, I am just not going to waste my time on this anymore. I will leave you with a printout of a test run on a new mailserver I'm building up right now, in fact, using an fxp card, to prove it's a not a stack problem. You can choose to believe it or you can choose to continue wasting your time chasing ghosts in the TP stack when the problem is the driver: I'm setting up test servers now, it's just taking time to get a good test environment up. I'll respond with actual numbers after testing, between autoneg and forced 100/full servers. I admit, the forced 100/full is because of ancient lore, particularly with cisco switches not always playing nice with autonegotiation, we've just always done it that way (until gbit), and never had any problems. Make absolutely sure to download the current catOS/IOS for your switches, older firmware in them had problems with certain network chipsets. Cisco got egg on it's face - the old IOS in the 2950's would not work with the new ethernet chipsets in the 1800/2800/3800 router series when they came out - among other things. The servers in question all do 150-200Mbit in production, no problem, it's just that any one flow can't do more than ~300KB/s cross country. Given that they're over 100Mbit, what ethernet card is recommended if em has problems? Your going to have to experiment, it's a crapshoot. I had a hell of a time with the bge adapter and 6.1 production, I produced a patch that helped, finally the bge author updated the driver with a more comprehensive fix. It works fine now but you must get the driver from CVS, the production 6.1 driver does not work. I also have an em card, but I didn't do significant testing with it after getting the bge fix. Our largest feed is 45Mbt and so I think it's pointless to plug a gigabit ethernet card into the network since a 10/100 card has plenty of capability to saturate our largest feed. None of our switches are gigabit and it is very unlikely that they will be upgraded in the near future. We do not do significant server-to-server data traffic, to be perfectly honest, I don't believe in it. I come from the school of you get 1 really big, powerful, expensive, reliable server that has enough power to do what you need, rather than a bunch of lame ones that are underpowered and try to cluster them. I've never had one of these fail in production, although I've seen a lot of clusters at customer sites that gave their admins a whole lot of grief. I only am dealing now with gigabit ethernet because I have to, since it's coming standard on all the new server hardware. And frankly I think it sucks, since I've seen lots of problems with gigE adapters at customer sites that were plugged into older switches. We haven't been bit by any of this yet - of course, we use 10/100 switches that were top-of-the-line switches during their day - but I've personally engineered 3 customer forklift upgrades to brand new top-of-the-line Cisco switches due to gigabit lan negotiation and throughput problems. Our customers have the dough to buy 80-100 ports of new Cisco switches, (of course they think they don't - but they do) wheres like most ISPs we don't. And, since we don't need it anyay, what's the point? FWIW, I am able to receive full speed on all of these servers. freebsd.org sends at 10Mbit, kernel.org at 20+. It's only sending speed that I have a problem with, and only with freebsd. My take on it is the gigabit ethernet chipset drivers are not completely debugged under FreeBSD at this time. Certainly, the Broadcom chipset is just getting there. The Intel chipsets usually lead the pack in support so you probably will get more traction on complaining to the em developer if you can demonstrate 100Mbt speeds on a fxp card, then 30Mbt speeds on an em card, in the same machine on the same network. FreeBSD tends to lag behind in the hardware support area. I'm sorry about that but you just have to accept it if your going to use FreeBSD. Ted ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: increasing transmit speeds in WAN setting?
- Original Message - From: Moses Leslie [EMAIL PROTECTED] To: Ted Mittelstaedt [EMAIL PROTECTED] Cc: freebsd-questions@freebsd.org Sent: Thursday, October 19, 2006 1:33 AM Subject: Re: increasing transmit speeds in WAN setting? Hi Ted, While I don't totally discount that possibility, I really don't think that's the case. I told you that you wouldn't believe me. We have over 500 servers, most of them running FreeBSD, and we've seen this happen in multiple cases on different hardware. Except all of them gigabit cards, right? So much for different hardware When it's linux, exact same hardware, exact same cables, this doesn't happen. It's an intel card gbit card, using the em driver. They're uplinked to Cisco 2948-getx switches, which are uplinked to 65xx's, which then go to 12xxx borders. There aren't any collision errors on the port at all: 24 totalCollisionCount= 0 25 lateCollisionCount = 0 26 singleCollisionFrames = 0 27 multipleCollisionFrames= 0 28 excessiveCollisionFrames = 0 and no real errors to speak of, period. The port is auto, since it needs to be to get gbit. All of the non-gbit servers we have are forced 100/full, all cisco switches, all intel 100/pro (fxp) drivers, they all show this same problem. Well right there you are doing things wrong. You should always set ethernet cards to auto. The only time you ever force 100/full or force anything, speed/duplex, is when your plugged into a hub that does NOT autoswitch. There's very few of them around that are 100base T, but there are some, and there's a lot more 10baseT stuff that wasn't autoswitching. Any halfway decent 100baseT hub will support nway autonegotiation and when you hard-code post speeds you will cause drops and speed loss. But, please don't take my word for it since you seem to like disbelieving me, just try it out yourself. Go to your fxp servers, login to your switches, set the switch port to the server to autonegotiation, on the server remove all the media options in /etc/rc.conf, shut down the server (you must power it down for the ports to switch into autonegotiation) and bring it up and you will see both sides negotiate to 100base T full, and a lot of your problems in throughput will disappear. Both the switch and the servermust be set to autonegotiation. If they don't autonegotiate to 100baseTfull, then you have a cable problem, simple as that. I've been doing ethernet since the late 80's and doing it professionally for a decade, and I've seen and use more different types of ethernet in my life than you will ever see in the rest of your career. The idea that your supposed to override the autonegotiation and hard code stuff originated from network admins who plugged early 100baseT stuff together then couldn't figure out why it didn't autonegotiate to 100baseT full. What they didn't realze is that the cabling they were using - CAT-3 mostly, or CAT-5 that had been incorrectly terminated with the wrong connectors, or wrong plugs, or wrong wiring pattern, or bad crimps because they were using stranded plugs on solid core wire, or some other such thing, what the real culprit, and the autonegotiation chips were in fact detecting the problem and trying to protect the network. Unfortunately, 90% of network admins out there don't know the first thing about layer-1, they assume the wiring contractors handle all that. The wiring contractors by contrast are mostly minimum-wage goobers who's heads are filled with a lot of rediculous nonsense about how Ethernet really works. If the server is a 4.9 server, I can get ~400KB/s. If it's 6.1, ~300KB/s. Linux 2.6, ~650KB/s, which is about what I'd expect given the latency and the default settings. All on the same hardware, same switches, same cables. The Linux device drivers are simply different than the FreeBSD drivers. I don't know how much more I can tell you this over and over. The em driver has got some problems, granted. But, this has absolutely nothing to do with the FreeBSD version or the TCP/IP stack. Until you do what I told you to do and properly setup and test under fxp0, I am just not going to waste my time on this anymore. I will leave you with a printout of a test run on a new mailserver I'm building up right now, in fact, using an fxp card, to prove it's a not a stack problem. You can choose to believe it or you can choose to continue wasting your time chasing ghosts in the TP stack when the problem is the driver: $ whoami tedm $ $ fetch ftp://ftp.freebsd.org/pub/FreeBSD/ls-lR.gz ls-lR.gz 100% of 18 MB 1057 kBps 00m00s $ $ ping ftp.freebsd.org PING ftp.freebsd.org (62.243.72.50): 56 data bytes 36 bytes from ge2-16.1000M.d5.opa.tdk.net (195.41.33.70): Communication prohibited by filter Vr HL TOS Len ID Flg off TTL Pro cks Src Dst 4 5 00 5400 82f2 0 33 01 6e38 65.75.206.14 62.243.72.50 ^C --- ftp.freebsd.org ping
Re: increasing transmit speeds in WAN setting?
On Thu, 19 Oct 2006, Ted Mittelstaedt wrote: Until you do what I told you to do and properly setup and test under fxp0, I am just not going to waste my time on this anymore. I will leave you with a printout of a test run on a new mailserver I'm building up right now, in fact, using an fxp card, to prove it's a not a stack problem. You can choose to believe it or you can choose to continue wasting your time chasing ghosts in the TP stack when the problem is the driver: I'm setting up test servers now, it's just taking time to get a good test environment up. I'll respond with actual numbers after testing, between autoneg and forced 100/full servers. I admit, the forced 100/full is because of ancient lore, particularly with cisco switches not always playing nice with autonegotiation, we've just always done it that way (until gbit), and never had any problems. The servers in question all do 150-200Mbit in production, no problem, it's just that any one flow can't do more than ~300KB/s cross country. Given that they're over 100Mbit, what ethernet card is recommended if em has problems? FWIW, I am able to receive full speed on all of these servers. freebsd.org sends at 10Mbit, kernel.org at 20+. It's only sending speed that I have a problem with, and only with freebsd. Thanks, Moses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: increasing transmit speeds in WAN setting?
Hi Moses, I know your not going to believe me but you are running into a driver bug of some kind. If you have a really high quality ethernet switch with full management in it you can probably see it - login to the switch and look at the port statistics. Cisco routers are designed to sense for this and you will see it in their logs, they will issue the error message late collissions or any decent hardware network sniffer will show it. The most common problem is the switch and network card aren't properly negotiating duplex. Another area is flow control on full duplex being messed up, this is particularly critical on gigabit E. The reason your getting good throughput on local connections is that the layer 1 is simply continuing to retransmit until the packet goes through, and the retransmissions are happening so fast that you don't realize it. That is also why latency is so heavily affecting it. You can try several things. First, temporarily try switching over to a 10/100 card like an Intel EtherExpress Pro/100 if you have a PCI slot in the server. If that works then your going to have to try replacing your switch. If you have a really good switch you can try hard coding it's ports speed and duplex and try the same on the server, and see if that does anything. You also should be aware that many of the smaller and cheaper gigabit switches do not have the ability to take sustained gigabit ethernet speeds with back-to-back packets, their internal processors aren't fast enough. Once more, this is a problem that won't show up on a local connection since the retransmissions are so fast. Ted - Original Message - From: Moses Leslie [EMAIL PROTECTED] To: freebsd-questions@freebsd.org Sent: Wednesday, October 18, 2006 10:31 PM Subject: increasing transmit speeds in WAN setting? Hi, We're running 6.1-R, and are having difficulty getting decent speeds as latency increases. The server is connected via gbit copper, and is gbit or better to the internet (depending on the path). For everything local, we're able to get what you'd expect (300+MBit without really any tuning). However, when the latency is 60-80ms (IE across the US), we're unable to get better than around 300KB/s. It appears to be possibly related to the tcp.inflight stuff, but disabling it or messing with some of the related sysctls doesn't appear to help much. Downloads often start quickly, but are then throttled back down to 300KB/s within 10 seconds or so. We've changed the hz (100 to 1), the net.inet.tcp.sendspace, kern.ipc.maxsockbuf, and tried different variations on the inflight tunables, but nothing has made a positive difference of more than ~20KB/s at best. If the server is running linux (2.6 kernel with default TCP settings), we can get much better speeds, 600-1000KB/s easily. If we were going for time/distance records, we would try changing around tcp settings on the client, but we're trying to maximize performance for standard surfers who wouldn't know how to do that, so we're looking for anything that is server side only. We've been searching high and low for any tuning ideas but aren't able to find anything that's made a difference. From looking at how the congestion stuff works in the source, it appears that something like: http://www.sigusr1.org/weblog/index.php?/categories/6-Hacks might be happening here, but we're kind of stabbing in the dark. Does anyone have any tuning ideas for 6.1 in a WAN setting? Thanks, Moses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED] ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
Re: increasing transmit speeds in WAN setting?
Hi Ted, While I don't totally discount that possibility, I really don't think that's the case. We have over 500 servers, most of them running FreeBSD, and we've seen this happen in multiple cases on different hardware. When it's linux, exact same hardware, exact same cables, this doesn't happen. It's an intel card gbit card, using the em driver. They're uplinked to Cisco 2948-getx switches, which are uplinked to 65xx's, which then go to 12xxx borders. There aren't any collision errors on the port at all: 24 totalCollisionCount= 0 25 lateCollisionCount = 0 26 singleCollisionFrames = 0 27 multipleCollisionFrames= 0 28 excessiveCollisionFrames = 0 and no real errors to speak of, period. The port is auto, since it needs to be to get gbit. All of the non-gbit servers we have are forced 100/full, all cisco switches, all intel 100/pro (fxp) drivers, they all show this same problem. If the server is a 4.9 server, I can get ~400KB/s. If it's 6.1, ~300KB/s. Linux 2.6, ~650KB/s, which is about what I'd expect given the latency and the default settings. All on the same hardware, same switches, same cables. The only error that increments at all is txQueueNotAvailable, which is to be expected as the BDP is figured out. I'm pretty sure that FreeBSD is throttling itself back when it shouldn't be. Thanks for the reply, Moses On Wed, 18 Oct 2006, Ted Mittelstaedt wrote: Hi Moses, I know your not going to believe me but you are running into a driver bug of some kind. If you have a really high quality ethernet switch with full management in it you can probably see it - login to the switch and look at the port statistics. Cisco routers are designed to sense for this and you will see it in their logs, they will issue the error message late collissions or any decent hardware network sniffer will show it. The most common problem is the switch and network card aren't properly negotiating duplex. Another area is flow control on full duplex being messed up, this is particularly critical on gigabit E. The reason your getting good throughput on local connections is that the layer 1 is simply continuing to retransmit until the packet goes through, and the retransmissions are happening so fast that you don't realize it. That is also why latency is so heavily affecting it. You can try several things. First, temporarily try switching over to a 10/100 card like an Intel EtherExpress Pro/100 if you have a PCI slot in the server. If that works then your going to have to try replacing your switch. If you have a really good switch you can try hard coding it's ports speed and duplex and try the same on the server, and see if that does anything. You also should be aware that many of the smaller and cheaper gigabit switches do not have the ability to take sustained gigabit ethernet speeds with back-to-back packets, their internal processors aren't fast enough. Once more, this is a problem that won't show up on a local connection since the retransmissions are so fast. Ted - Original Message - From: Moses Leslie [EMAIL PROTECTED] To: freebsd-questions@freebsd.org Sent: Wednesday, October 18, 2006 10:31 PM Subject: increasing transmit speeds in WAN setting? Hi, We're running 6.1-R, and are having difficulty getting decent speeds as latency increases. The server is connected via gbit copper, and is gbit or better to the internet (depending on the path). For everything local, we're able to get what you'd expect (300+MBit without really any tuning). However, when the latency is 60-80ms (IE across the US), we're unable to get better than around 300KB/s. It appears to be possibly related to the tcp.inflight stuff, but disabling it or messing with some of the related sysctls doesn't appear to help much. Downloads often start quickly, but are then throttled back down to 300KB/s within 10 seconds or so. We've changed the hz (100 to 1), the net.inet.tcp.sendspace, kern.ipc.maxsockbuf, and tried different variations on the inflight tunables, but nothing has made a positive difference of more than ~20KB/s at best. If the server is running linux (2.6 kernel with default TCP settings), we can get much better speeds, 600-1000KB/s easily. If we were going for time/distance records, we would try changing around tcp settings on the client, but we're trying to maximize performance for standard surfers who wouldn't know how to do that, so we're looking for anything that is server side only. We've been searching high and low for any tuning ideas but aren't able to find anything that's made a difference. From looking at how the congestion stuff works in the source, it appears that something like: http://www.sigusr1.org/weblog/index.php?/categories/6-Hacks might be happening here, but we're kind of stabbing in the dark. Does anyone have any tuning ideas for 6.1 in a WAN setting? Thanks
Re: increasing transmit speeds in WAN setting?
One other point of data is tha that this is a per-flow limit. If we do 10 wget's or whatever, it will be approximately 10x the data rate, IE 30MBit vs 3Mbit. Thanks, Moses On Wed, 18 Oct 2006, Moses Leslie wrote: Hi, We're running 6.1-R, and are having difficulty getting decent speeds as latency increases. The server is connected via gbit copper, and is gbit or better to the internet (depending on the path). For everything local, we're able to get what you'd expect (300+MBit without really any tuning). However, when the latency is 60-80ms (IE across the US), we're unable to get better than around 300KB/s. It appears to be possibly related to the tcp.inflight stuff, but disabling it or messing with some of the related sysctls doesn't appear to help much. Downloads often start quickly, but are then throttled back down to 300KB/s within 10 seconds or so. We've changed the hz (100 to 1), the net.inet.tcp.sendspace, kern.ipc.maxsockbuf, and tried different variations on the inflight tunables, but nothing has made a positive difference of more than ~20KB/s at best. If the server is running linux (2.6 kernel with default TCP settings), we can get much better speeds, 600-1000KB/s easily. If we were going for time/distance records, we would try changing around tcp settings on the client, but we're trying to maximize performance for standard surfers who wouldn't know how to do that, so we're looking for anything that is server side only. We've been searching high and low for any tuning ideas but aren't able to find anything that's made a difference. From looking at how the congestion stuff works in the source, it appears that something like: http://www.sigusr1.org/weblog/index.php?/categories/6-Hacks might be happening here, but we're kind of stabbing in the dark. Does anyone have any tuning ideas for 6.1 in a WAN setting? Thanks, Moses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
increasing transmit speeds in WAN setting?
Hi, We're running 6.1-R, and are having difficulty getting decent speeds as latency increases. The server is connected via gbit copper, and is gbit or better to the internet (depending on the path). For everything local, we're able to get what you'd expect (300+MBit without really any tuning). However, when the latency is 60-80ms (IE across the US), we're unable to get better than around 300KB/s. It appears to be possibly related to the tcp.inflight stuff, but disabling it or messing with some of the related sysctls doesn't appear to help much. Downloads often start quickly, but are then throttled back down to 300KB/s within 10 seconds or so. We've changed the hz (100 to 1), the net.inet.tcp.sendspace, kern.ipc.maxsockbuf, and tried different variations on the inflight tunables, but nothing has made a positive difference of more than ~20KB/s at best. If the server is running linux (2.6 kernel with default TCP settings), we can get much better speeds, 600-1000KB/s easily. If we were going for time/distance records, we would try changing around tcp settings on the client, but we're trying to maximize performance for standard surfers who wouldn't know how to do that, so we're looking for anything that is server side only. We've been searching high and low for any tuning ideas but aren't able to find anything that's made a difference. From looking at how the congestion stuff works in the source, it appears that something like: http://www.sigusr1.org/weblog/index.php?/categories/6-Hacks might be happening here, but we're kind of stabbing in the dark. Does anyone have any tuning ideas for 6.1 in a WAN setting? Thanks, Moses ___ freebsd-questions@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-questions To unsubscribe, send any mail to [EMAIL PROTECTED]
