Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: : :No switching infrastructure. It's a 10mb/s half duplex ethernet :network, with two hubs between the two machines. : :Joe I think there may be a problem with your hub setup (e.g. exceeding the hub count or end-to-end length limitations) that is either resulting in packet loss, or the packet burst is locking up the ethernet long enough to cause a timeout. The problem looks very much like an unrecognized collision to me. I recommend replacing the hubs with switches. The whole topology will be happier. Or maybe just with better hubs. I've seen cheap hubs that started losing packets when the load went above ~5MB/s. -SB To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
In message: [EMAIL PROTECTED] Matthew Dillon [EMAIL PROTECTED] writes: : I think that has been fixed. Try it. It doesn't lag for me. The : turn-around echo of the keystroke should be pushed out instantly. Yes. Keep in mind that you only need to enable this when you have unacked data. Otherwise, NODELAY doesn't buy you anything. I routinely go over non-NODELAY links and haven't seen a huge problem except when modems were involved that liked to freak out. Warner To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Louis A. Mamakos wrote: Disabling Nagle's algorithm for no good reason has very poor scaling behavior. This is what happens when TCP_NODELAY is enabled on a socket. Disabling Nagle's algorithm for a good reason would still result in the observed failure, however. If you look at the work function for most network elements, the part that runs out of gas first is per-packet forwarding performance. Sure, you need to have adequate bus bandwidth to move stuff through a box, but it's performing per-packet forwarding operations and policy which is the resource that's most difficult to make more of. I think this is true for toy routers based on PC platform as well as high-end boxes like the Cisco 12000 series. Juniper managed adequate forwarding performance using specialized ASIC implementions in the forwarding path. Of this statement, I'm sure; in my day job at UUNET, I talk to all the major backbone router vendors, and forwarding performance (and also reasonable routing protocol implementions) is a show-stopper requirement they labor mightily over. PCI is sufficient to keep a Gigabit interface saturated, even without going to jumbograms. I have personally saturated such an interface. PCI-X will scale to 8 Gigabits. So here was have a mechanism with wonderful properties - it's a trivial yet clever implementation of a self tuning mechanism to prevent tinygrams from being generated by a TCP without all manner of complicated timers. It give great performance on LAN and other high-speed interconnects where remote echo type applications are demanding, yet over long delay paths where remote echo is gonna suck no matter what you do, it automatically aggregates packets. As a bandwidth provider, UUNET is more concerned with aggregate throughput; this means it cares more about the moving average of packets getting through than it does about *my* packets getting through. When it comes to a conflict of interest, you will understand my preferences are for my own interests... 8-). Nagle's algorithm and Van Jacobson's slow-start algorithm allowed the Internet to survive over congested paths. And they did so with a bunch of self-tuning behavior independent of the bandwidth*delay product of the path the connection was running over. It was and is amazing stuff. Yes, I'm well aware of bandwidth delay product calculations; it's the primary mechanism behind the rate halving algorithm I keep pointing to (Hoe, Jacobson). 8-). It's also the primary limitation on connection speed (remember that it was my FreeBSD machine that was able to get to 1.6M connections, with standard sockets). Likewise, the original problem in this thread is likely caused by some part of the USB Ethernet implementation having inadequate per-packet resources. It's probably not about the number of bytes, but the number of transactions. You see here a modern reimplementation of essentially the same problem that the 3COM 3C501 ISA ethernet card had 15 years ago - back to back packets were consistantly dropped because of the poor per-packet buffering implementation. It was absolutely repeatable. Clearly. I think that's well established, since no one has squawked about the FreeBSD USB driver or the PC USB hardware being slower than the dongle USB hardware... Sure, it's legal to generate streams of tinygrams and not use Nagle's algorithm to aggregate the sender's traffic, but it's just plain rude and on low bandwidth links, it sucks because of all the extra 40 byte headers you're carrying around. I understand this. But Nagle is not the only mechanism which would fix the problem. Given that it's *intentionally* possible *and permitted* to turn Nagle off (via TCP_NODELAY), it makes sense to look at another mechanism that is not succeptible to being turned off. I'm sure TCP_NODELAY got added because it sounds REALLY C00L to make the interactive thing go better. But clearly people don't understand the impact of turning on the cleverly named option and how it probably doesn't really improve things. I'm pretty sure it got added to address interactive response on intrinsically small packets; telnet was probably the number one reason, but interactive responsiveness of small requests over TCP, such as non-pipelined HTTP requests, SMTP server responses, FTP control channel traffic, and other protocols benefit significantly from turning Nagle off. Nagle almost suggests it, in the original paper. Since OpenSSH incredibly bloats payload, it's much less necessary to get performance as it would be out of Telnet, though you will likely see least common multiples of the MTU cause occasionaly burstiness. Probably in this case, it would make sense to increase the decay rate of the timer based on the amount of data that is pending, relative to the MTU: defeat Nagle when the pipe is full, and partially defeat it down to, say, the pipe being half full. Thus large streams not ending on MTU boundaries would not
Re: FreeBSD performing worse than Linux?
M. Warner Losh wrote: In message: [EMAIL PROTECTED] Matthew Dillon [EMAIL PROTECTED] writes: : I think that has been fixed. Try it. It doesn't lag for me. The : turn-around echo of the keystroke should be pushed out instantly. Yes. Keep in mind that you only need to enable this when you have unacked data. Otherwise, NODELAY doesn't buy you anything. I routinely go over non-NODELAY links and haven't seen a huge problem except when modems were involved that liked to freak out. My experience agrees with David's. It makes sense to set TCP_NODELAY on the client, still, if the payload is still small enough after encryption. On the server side, it makes less sense. The server side TCP_NODELAY is the problem in this case, since it is the client, not the server, which is on the slow side of a USB dongle, FWIW. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Remember also that Matt recently shot Reno for performance reasons Actually, it was turned back on less than 72 hours later, when he found/fixed a bug in NewReno. It was only off for a little bit, and only in -stable. , when compared to Linux, when he should probably have simply cranked initial window size to 3 (Jacobson) and added piggy-back ACKs (Mogul). See above. You need to pay more attention to what's going on. :) Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: new-reno was artifically limiting the max number of in-transit packets to 4. This is probably why the USB ethernet worked with 4.3, but it destroyed TCP performance for everything else and was removed. I don't think there is kernel solution to the USB problems, short of fixing the driver (if that's even possible). The packets are being sent from the server and the only thing the client has control over is the receive window advertisement. You could artifically reduce the window advertisement on the client to deal with lots of small packets but it would destroy performance for larger packets / data streams. Cap the window size via the driver. This will probably require another attribute on all drivers, indicating max allowable window size (0 could mean any, and be used as the default so that we don't end up flaking out someone talking to Mars). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:Cap the window size via the driver. This will probably require :another attribute on all drivers, indicating max allowable window :size (0 could mean any, and be used as the default so that we :don't end up flaking out someone talking to Mars). : :-- Terry It won't help. The problem is that the server is sending more data then the client USB adapter can handle. If the client limits its window size with the expectation that the server will send it small packets, then any *other* transfer that uses larger packets will have terrible performance. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: :Cap the window size via the driver. This will probably require :another attribute on all drivers, indicating max allowable window :size (0 could mean any, and be used as the default so that we :don't end up flaking out someone talking to Mars). It won't help. The problem is that the server is sending more data then the client USB adapter can handle. If the client limits its window size with the expectation that the server will send it small packets, then any *other* transfer that uses larger packets will have terrible performance. I'm talking about capping the maximum negotiable window size over the USB adapter... How can this cause problems, since (1) the only thing we are talking about is window size, not packet size (MTU), and (2) the only thing it will effect is the TCP stream post negotiation, so it should not result in a slowdown for any virtual circuit other than those that *NEED* a slowdown? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:I'm talking about capping the maximum negotiable window size :over the USB adapter... : :How can this cause problems, since (1) the only thing we are talking :about is window size, not packet size (MTU), and (2) the only thing :it will effect is the TCP stream post negotiation, so it should not :result in a slowdown for any virtual circuit other than those that :*NEED* a slowdown? : :-- Terry If you look at the tcp dump Josef sent, the server was sending 32-byte (data) packets over TCP. It only took 8 packets or so to blow up the USB adapter. 8x32 = 256 bytes. So unless you intend to cap the window size at, say, 128 bytes, this will solve the ssh issue but destroy performance for everything else going over the USB ethernet. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
the driver should return ENOBUFS and TCP should repect that.. I think it already does in fact.. On Mon, 31 Dec 2001, Matthew Dillon wrote: :I'm talking about capping the maximum negotiable window size :over the USB adapter... : :How can this cause problems, since (1) the only thing we are talking :about is window size, not packet size (MTU), and (2) the only thing :it will effect is the TCP stream post negotiation, so it should not :result in a slowdown for any virtual circuit other than those that :*NEED* a slowdown? : :-- Terry If you look at the tcp dump Josef sent, the server was sending 32-byte (data) packets over TCP. It only took 8 packets or so to blow up the USB adapter. 8x32 = 256 bytes. So unless you intend to cap the window size at, say, 128 bytes, this will solve the ssh issue but destroy performance for everything else going over the USB ethernet. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
: :the driver should return ENOBUFS :and TCP should repect that.. :I think it already does in fact.. How does the client's driver returning ENOBUFS prevent the server from sending too many packets? -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:Matt: make up your mind. : :Either: : :1) The server is sending too many packets : :OR : :2) The server is NOT sending too many packets : :The correct way to deal with #1 is to negotiate a smaller window size :so that too many packets are not sent. : :Actually, the PSC rate-halving algorithm would probably fix this :right up... : :-- Terry Terry, I don't think you quite understand the problem. From the server's point of view NOTHING IS WRONG. Any non-USB-ethernet client would have no problem whatsoever with the large number of small packets that the server is sending, nor can the server just arbitrarily decide that 8 packets is too much, because doing so screws up bandwidth over long-haul links. The server can not arbitrarily limit its window size because the problem only occurs with small packets. So how is the server supposed to figure out the difference between between normal packet loss and packet loss due to the client having a USB adapter and not being able to handle lots of tiny packets? New-Reno might be able to help but it would have to be hacked up a bit to send a double ack to the server to force the server to reduce its congestion window. New-Reno is a mess and I am not volunteering to hack it up more. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
my stupid :-) On Mon, 31 Dec 2001, Matthew Dillon wrote: : :the driver should return ENOBUFS :and TCP should repect that.. :I think it already does in fact.. How does the client's driver returning ENOBUFS prevent the server from sending too many packets? -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Mon, 31 Dec 2001, Terry Lambert wrote: Matthew Dillon wrote: :the driver should return ENOBUFS :and TCP should repect that.. :I think it already does in fact.. How does the client's driver returning ENOBUFS prevent the server from sending too many packets? Matt: make up your mind. Either: 1)The server is sending too many packets OR 2)The server is NOT sending too many packets The correct way to deal with #1 is to negotiate a smaller window size so that too many packets are not sent. Windows are set in BYTES not packets.. what IS set in packets is the backoff winsow size (I forget it's real name) in the slow-start code.. there may be something you could do there... Actually, the PSC rate-halving algorithm would probably fix this right up... -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: Terry, I don't think you quite understand the problem. From the server's point of view NOTHING IS WRONG. Any non-USB-ethernet client would have no problem whatsoever with the large number of small packets that the server is sending, nor can the server just arbitrarily decide that 8 packets is too much, because doing so screws up bandwidth over long-haul links. The server can not arbitrarily limit its window size because the problem only occurs with small packets. So how is the server supposed to figure out the difference between between normal packet loss and packet loss due to the client having a USB adapter and not being able to handle lots of tiny packets? Uh, Matt, the window size is a negotiated value between the server and the client. What the server wants is irrelevant, if the client wants a smaller window, since the value is MIN(server_wants,client_wants). So if the problem is that the FreeBSD USB-ethernet connected client can't handle that many outstanding packets, then the place to fix it is _obviously_ the FreeBSD USB-ethernet driver. Julian did similar work already in the InterJet in order to control the advertised window size on the client, and therefore do what the AltQ code is supposed to do, but without congesting the intermediate router. New-Reno might be able to help but it would have to be hacked up a bit to send a double ack to the server to force the server to reduce its congestion window. New-Reno is a mess and I am not volunteering to hack it up more. The Rate-Halving algorithm is used with FACK in order to halve the window size in the presence of failures (like the one being seen here). Again, the place that this should happen is on the client, not the server. I don't understand why you think this is a server problem, when it is not the server hardware where the failure is occuring? -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Terry, I give up. Maybe if you actually tried to go in and fix it you would see what I'm talking about. For your information: * Julian's work has nothing to do with this particular problem. It has to do with constricted bandwidth. This issue has nothing to do with USB's bandwidth limitations. * The window size is not negotiated between client and server (other then whether window scaling is used or not, which is non-applicable). Each side's TCP protocol receiver controls the window size it advertises. * The window size has no bearing on the number of in-transit packets, unless you make assumptions in regards to the packet size. That's the crux of the problem faced by the receiver in this case. * If you think reducing the receive window on the fly is easy, try it and you will find out that it isn't as easy as you might have thought. * And if you think you can handle that, now try figuring out, in the receiver, how to dynamically increase and reduce the window based on the size of the packets being received and try to discern the difference between packet loss and a driver problem, to then limit the number of in-transit packets (through a massive window reduction). Good luck! * I never said the server was the problem. To the contrary, I've been saying that the client is the problem.. USB ethernet is broken, period. As I have said, it may be possible to make new-reno on the client (receiver) side to cause the transmit side to close its congestion window (and to keep it closed, or fairly closed). But I aint gonna be the one to try to do it. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: Terry, I give up. Maybe if you actually tried to go in and fix it you would see what I'm talking about. For your information: I don't have a USB/Ethernet adapter. * Julian's work has nothing to do with this particular problem. It has to do with constricted bandwidth. This issue has nothing to do with USB's bandwidth limitations. The specific work in question is rate limiting the *other end* of a TCP connection. This would let a client with a problem like this *slow the server down*, without any code changes on the server, and without buffer congestion on the server. You have not seen this work, I think. * The window size is not negotiated between client and server (other then whether window scaling is used or not, which is non-applicable). Each side's TCP protocol receiver controls the window size it advertises. So you are saying that if the client advertised a smaller TCP window, then the server would have less simultaneously oustanding packets in flight before an ACK was required. I think you and I are just using different definitions of negotiated: the client controls the transmit window which is available to the server. So we are agreed: If the client with the USB bogosity were to advertise a smaller window, then the server would have less packets in flight; and since the problem appears (to me) to be that there are more packets in flight from the server than the USB-ethernet dongle is capable of buffering to send down the slow (USB) side of the link, then this would prevent the server from buffer-overflowing the dongle into losing packets. It seems to me that this is a clear case of impedence mismatch, with a fast link trying to jam more packets down a slow link than is possible, with the ACKs being sent by the dongle, rather than delayed until it has sufficient buffer space available. * The window size has no bearing on the number of in-transit packets, unless you make assumptions in regards to the packet size. That's the crux of the problem faced by the receiver in this case. This is why I suggested that the PSC rate-halving algorithm would probably be useful. The real problem here is that more data is being sent to the dongle than it is able to forward. The packet trace bears this conclusion out. * If you think reducing the receive window on the fly is easy, try it and you will find out that it isn't as easy as you might have thought. Both Julian's code and the PSC code do this. I wasn't suggesting writing the code from scratch. However, I have to say that I've implicitly done this sort of limitation in a commercial product, before, anolg with Jeffrey Hsu, by controlling transmit and receive window sizes in a proxy server. Clearly, a web server on a 1G link talking to a client on the other end of a 28.8k link would quickly overrun a proxy server's transmit buffers, otherwise, as the received data is shoved out at a much slower rate. This failure mode is consistant with the use of a much slower USB link on the other side of a USB-ethernet dongle. The problem here is that the don'e itself is not going to do the necessary flow rate limiting (it doesn't, or this discussion would have never happened in the first place), so you have to trigger it as a secondary effect, like the PSC rate halving, or the code Julian did to force the advertised window sizes smaller over time. Julian's code was exactly analogous to the dongle situation, except the dongle was a CISCO router on the other end of a slow link from an InterJet, where we wanted a single flow to not monopolize all the transmit buffers on the router, so that we would stil be able to get data through. This basically meant controlling the amount of the pool taken up, times the pool retention time, on the router, for any given set of flows. As a side effect, you could intentionally reduce the limit on the advertised window size, until the flow rate dropped. This is basically the balance the TCP Rate Halving code seeks to achieve (the balance between buffer congestion losses and the overall throughput). Please see: http://www.psc.edu/networking/rate_halving.html * And if you think you can handle that, now try figuring out, in the receiver, how to dynamically increase and reduce the window based on the size of the packets being received and try to discern the difference between packet loss and a driver problem, to then limit the number of in-transit packets (through a massive window reduction). Good luck! Julian already did this, as I stated. * I never said the server was the problem. To the contrary, I've been saying that the client is the problem.. USB ethernet is broken, period. As I have said, it may be possible to make new-reno on the client (receiver) side to cause the transmit side to close its congestion window
Re: FreeBSD performing worse than Linux?
An underlying issue here is why applications decide to set TCP_NODELAY options on sockets, rather than just letting Nagle's algorithm do the right thing. I recall some handwaving about this in the X server some years ago to make mouse movements smoother. For the problem at hand, if both the client and server machines didn't do TCP_NODLEAY, then there'd only be one packet smaller than the TCP MSS in flight between the transmitter and receiver at any one time. I think that poking OpenSSH to not set the TCP_NODELAY option fixed this problem. I was just pondering the TCP implementation in 4.5-PRERELEASE, and it doesn't look like there's any explicit delay after a write going on, other than Nagle's algorithm, in the TCP packetization code. So setting TCP_NODELAY is almost certain the Wrong Thing for most applications to do. Perhaps there ought to be a warning in the man page about being a poor network citizen, flooding the Internet with tinygrams and otherwise making the performance of your application generally suck. louie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
In a message written on Mon, Dec 31, 2001 at 02:44:42PM -0800, Matthew Dillon wrote: * I never said the server was the problem. To the contrary, I've been saying that the client is the problem.. USB ethernet is broken, period. I think several people are trying to say the same thing off this point but not saying it the same way, so: * Hacking TCP (or pretty much anything else) is the _wrong_ solution to this problem. The solution is to fix USB ethernet and/or not use it. * Since this is a fairly well defined, easy to reproduce packet loss situation that TCP seems to not handle particularly well, it might be worth using it to investigate if there is a generic TCP improvment that could be made. Fair enough? -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Louis A. Mamakos wrote: An underlying issue here is why applications decide to set TCP_NODELAY options on sockets, rather than just letting Nagle's algorithm do the right thing. I recall some handwaving about this in the X server some years ago to make mouse movements smoother. OK, I've been making an assumption here that is perhaps incorrect, but is consistant with the traces posted so far: I've assumed that the data is lost between the ethernet side of the dongle and the USB side of the dongle. If the FreeBSD USB driver is really a dog (I have a hard time buying that) or the dongle USB hardware is superior to that in the PC being used (I have a hard time buying that one, either), then it's just possible that the data losses are FreeBSD's, and not a result of dongle buffer overrun. You'd need a breakout box for USB, and a USB system without the problem (Linux?) to prove it via dumps, though. That said: There is no problem with turning on TCP_NODELAY. If anything, the resulting data sent through the dongle would be *less* bursty (being application paced), and tend to be *less* likely to fill up any pool in the middle to the point of overflow. This really has all the earmarks of a pool congestion (not a link congestion) problem in the dongle, to my mind. For the problem at hand, if both the client and server machines didn't do TCP_NODLEAY, then there'd only be one packet smaller than the TCP MSS in flight between the transmitter and receiver at any one time. I think that poking OpenSSH to not set the TCP_NODELAY option fixed this problem. I think it masked it. If you had a less CPU-intensive test application, I think you would see that you can still overrun the buffer. The masking is a natural effect of decreasing the overall payload encapsulation overhead. But increase the amount of payload, and you should still see the problem rear its ugly head, since the link rate differential is so large on either side of the dongle. A fugly way of handling the problem would be to initiate a flow control mechanism between the dongle and the FreeBSD ethernet driver. This is tantamount to the hardware flow control required for the MNP modem knock-offs by Microcomm competitors who used flow control as a means of keeping the pool size manageable, so that they could really cheap out on putting RAM in the modem, and undercut Microcomm's prices. For this to work, you'd have to have cooperative firmware in the dongles; I think that taking that approach is a lost cause, unless we are downloading our own firmware. I think we will have to tackle it another way. Teaching the driver about TCP windows, and delaying transmissions until 50% of the receive window is satisfied would probably work, but is a very unsatisfactory approach (even though we know that transmit over the link is not going to have the same problem as receive, due to the speed differential across the dongle). -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:An underlying issue here is why applications decide to set TCP_NODELAY :options on sockets, rather than just letting Nagle's algorithm do :the right thing. I recall some handwaving about this in the X server :some years ago to make mouse movements smoother. : :For the problem at hand, if both the client and server machines didn't :do TCP_NODLEAY, then there'd only be one packet smaller than the :TCP MSS in flight between the transmitter and receiver at any one :time. I think that poking OpenSSH to not set the TCP_NODELAY option :fixed this problem. : :I was just pondering the TCP implementation in 4.5-PRERELEASE, and :it doesn't look like there's any explicit delay after a write going :on, other than Nagle's algorithm, in the TCP packetization code. So :setting TCP_NODELAY is almost certain the Wrong Thing for most :applications to do. Perhaps there ought to be a warning in the :man page about being a poor network citizen, flooding the Internet :with tinygrams and otherwise making the performance of your application :generally suck. : :louie Yes, you are correct. There is no real reason for ssh to set TCP_NODELAY on FreeBSD and, in fact, I believe it didn't used to. We should just turn it off. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Matthew Dillon wrote: Yes, you are correct. There is no real reason for ssh to set TCP_NODELAY on FreeBSD and, in fact, I believe it didn't used to. We should just turn it off. FWIW, I agree that it should not be set. Setting socket options has the unfortunate side effect of grabbing more per connection memory than would otherwise be necessary. Still, it's legal to set the option, even if it's not really a sane thing to do in many cases, so disallowing it is not a fix for anything. -- Terry To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Disabling Nagle's algorithm for no good reason has very poor scaling behavior. This is what happens when TCP_NODELAY is enabled on a socket. If you look at the work function for most network elements, the part that runs out of gas first is per-packet forwarding performance. Sure, you need to have adequate bus bandwidth to move stuff through a box, but it's performing per-packet forwarding operations and policy which is the resource that's most difficult to make more of. I think this is true for toy routers based on PC platform as well as high-end boxes like the Cisco 12000 series. Juniper managed adequate forwarding performance using specialized ASIC implementions in the forwarding path. Of this statement, I'm sure; in my day job at UUNET, I talk to all the major backbone router vendors, and forwarding performance (and also reasonable routing protocol implementions) is a show-stopper requirement they labor mightily over. So here was have a mechanism with wonderful properties - it's a trivial yet clever implementation of a self tuning mechanism to prevent tinygrams from being generated by a TCP without all manner of complicated timers. It give great performance on LAN and other high-speed interconnects where remote echo type applications are demanding, yet over long delay paths where remote echo is gonna suck no matter what you do, it automatically aggregates packets. Nagle's algorithm and Van Jacobson's slow-start algorithm allowed the Internet to survive over congested paths. And they did so with a bunch of self-tuning behavior independent of the bandwidth*delay product of the path the connection was running over. It was and is amazing stuff. Likewise, the original problem in this thread is likely caused by some part of the USB Ethernet implementation having inadequate per-packet resources. It's probably not about the number of bytes, but the number of transactions. You see here a modern reimplementation of essentially the same problem that the 3COM 3C501 ISA ethernet card had 15 years ago - back to back packets were consistantly dropped because of the poor per-packet buffering implementation. It was absolutely repeatable. Sure, it's legal to generate streams of tinygrams and not use Nagle's algorithm to aggregate the sender's traffic, but it's just plain rude and on low bandwidth links, it sucks because of all the extra 40 byte headers you're carrying around. I'm sure TCP_NODELAY got added because it sounds REALLY C00L to make the interactive thing go better. But clearly people don't understand the impact of turning on the cleverly named option and how it probably doesn't really improve things. louie To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Yes, you are correct. There is no real reason for ssh to set TCP_NODELAY on FreeBSD and, in fact, I believe it didn't used to. We should just turn it off. Nagle + delayed acks don't really get along all that well, and often result in character echoes lagging .25-.5 seconds behind the keystrokes (due to round trip + 200ms). This really makes editing files and other interactive jobs rather painful. -DG David Greenman Co-founder, The FreeBSD Project - http://www.freebsd.org President, TeraSolutions, Inc. - http://www.terasolutions.com President, Download Technologies, Inc. - http://www.downloadtech.com Pave the road of life with opportunities. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:TCP_NODELAY on FreeBSD and, in fact, I believe it didn't used to. :We should just turn it off. : : Nagle + delayed acks don't really get along all that well, and often :result in character echoes lagging .25-.5 seconds behind the keystrokes (due :to round trip + 200ms). This really makes editing files and other interactive :jobs rather painful. : :-DG : :David Greenman I think that has been fixed. Try it. It doesn't lag for me. The turn-around echo of the keystroke should be pushed out instantly. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sat, Dec 29, 2001 at 10:44:31PM -0800, Matthew Dillon wrote: Ok, there's packet loss. From this extract you can see that the client receives through sequence 641, then the next packet it receives starts at sequence 993. 15:28:09.879928 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 609:641(32) ack 64 win 33304 nop,nop,timestamp 152269835 86 8399 (DF) [tos 0x10] 15:28:09.881926 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 993:1025(32) ack 64 win 33304 nop,nop,timestamp 152269835 8 68402 (DF) [tos 0x10] If you look at the server you can see the (tiny) packets being transmitted: 15:28:18.255648 transwarp.telnet genius.kpop: P 609:641(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:18.255775 transwarp.telnet genius.kpop: P 641:673(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:18.255864 genius.kpop transwarp.telnet: . ack 97 win 33288 nop,nop,timestamp 868402 152269835 (DF) [tos 0x10] 15:28:18.255955 transwarp.telnet genius.kpop: P 673:705(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256084 transwarp.telnet genius.kpop: P 705:737(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256223 transwarp.telnet genius.kpop: P 737:769(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256351 transwarp.telnet genius.kpop: P 769:801(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256479 transwarp.telnet genius.kpop: P 801:833(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256607 transwarp.telnet genius.kpop: P 833:865(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256734 transwarp.telnet genius.kpop: P 865:897(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256884 transwarp.telnet genius.kpop: P 897:929(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.257011 transwarp.telnet genius.kpop: P 929:961(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.257138 transwarp.telnet genius.kpop: P 961:993(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.258068 transwarp.telnet genius.kpop: P 993:1025(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.258221 transwarp.telnet genius.kpop: P 1025:1057(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10 The first one is through 641. The server then sends 11 packets (641 through 993) that the client misses completely. The next packet the client sees is the 993:1025 packet. So there is nothing wrong with the TCP protocol. The question now is whether this is packet loss due to the physical layer or whether it is a queueing problem. What kind of network cards do these machines have and what kind of switching infrastructure is between them? No switching infrastructure. It's a 10mb/s half duplex ethernet network, with two hubs between the two machines. One thing that doesn't make sense is that the client is responding to each packet with its own ack but the server doesn't see the acks until it finishes transmitting a big stream of data packets. This implies half-duplex operation. Yes, it is half-duplex. Joe msg30559/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
: :No switching infrastructure. It's a 10mb/s half duplex ethernet :network, with two hubs between the two machines. : :... : :Yes, it is half-duplex. : :Joe I think there may be a problem with your hub setup (e.g. exceeding the hub count or end-to-end length limitations) that is either resulting in packet loss, or the packet burst is locking up the ethernet long enough to cause a timeout. The problem looks very much like an unrecognized collision to me. I recommend replacing the hubs with switches. The whole topology will be happier. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sun, Dec 30, 2001 at 11:17:39AM -0800, Matthew Dillon wrote: I think there may be a problem with your hub setup (e.g. exceeding the hub count or end-to-end length limitations) that is either resulting in packet loss, or the packet burst is locking up the ethernet long enough to cause a timeout. The problem looks very much like an unrecognized collision to me. I'll try cutting some of the intermediate infrastructure out and see if that helps. I recommend replacing the hubs with switches. The whole topology will be happier. Of course. I've not found a well priced switch though. (on a budget). Thanks for helping me with this. I'll get back once I've reduced the network to something smaller. Joe msg30562/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
:I'll try cutting some of the intermediate infrastructure out and see if :that helps. : : I recommend replacing the hubs with switches. The whole topology will : be happier. : :Of course. I've not found a well priced switch though. (on a budget). : :Thanks for helping me with this. I'll get back once I've reduced the :network to something smaller. : :Joe I use cheap 8 and 16-port netgear switches mostly. But even a little 5-port switch may work for you, at least to start, to split your one logical segment into five. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sun, Dec 30, 2001 at 11:32:34AM -0800, Matthew Dillon wrote: :I'll try cutting some of the intermediate infrastructure out and see if :that helps. : : I recommend replacing the hubs with switches. The whole topology will : be happier. : :Of course. I've not found a well priced switch though. (on a budget). : :Thanks for helping me with this. I'll get back once I've reduced the :network to something smaller. : :Joe I use cheap 8 and 16-port netgear switches mostly. But even a little 5-port switch may work for you, at least to start, to split your one logical segment into five. There's not a lot of traffic on this side of the network, so hubs aren't too bad. Ok, this is the topology: laptop [upstairs 10base hub]-[Downstairs 10base hub] -| | DSL ~~[Alcatel Router] - [FXP server FXP] -| I get the same problem if I plug the laptop directly into the downstairs hub, and also if I plug it into the router (which has four hub ports on it). The server is configured as an ethernet bridging firewall. I think that it is extremely unlikely that the network infrastructure is failing in the same way when I plug into three different parts of it. Also, I didn't used to have this problem even with the came cabling. Joe msg30564/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
:=20 : I use cheap 8 and 16-port netgear switches mostly. But even a little : 5-port switch may work for you, at least to start, to split your one : logical segment into five. : :There's not a lot of traffic on this side of the network, so hubs aren't :too bad. : :Ok, this is the topology: : : :laptop [upstairs 10base hub]-[Downstairs 10base hub] -| : | :DSL ~~[Alcatel Router] - [FXP server FXP] -| : :I get the same problem if I plug the laptop directly into the downstairs :hub, and also if I plug it into the router (which has four hub ports on :it). The server is configured as an ethernet bridging firewall. : :I think that it is extremely unlikely that the network infrastructure is :failing in the same way when I plug into three different parts of it. :Also, I didn't used to have this problem even with the came cabling. : :Joe :... Well, the amount of traffic isn't the problem. You have three hubs, one of them being the Alcatel. Hmm. It's within what you are allowed to do with HUBs but I don't trust it. I would either replace the downstairs hub with a switch, or I would do this: [upstairs]-[downstairs-small-switch] | | | | down alcatel stairsrouter... hub | (other stuff) It would also be a good idea to test the laptop more directly, by connecting the FXP server to a switch and then connecting the laptop directly to the same switch. The problem with HUBs is collision detection latency, but even a collision occuring locally (i.e. between two downstairs machines) can screw up the whole net if the hub configuration goes out of spec. Collision detection problems often rear their ugly heads with small packets, because the ethernet controllers don't even get 'late collision' errors. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sun, Dec 30, 2001 at 12:38:59PM -0800, Matthew Dillon wrote: Well, the amount of traffic isn't the problem. You have three hubs, one of them being the Alcatel. Hmm. It's within what you are allowed to do with HUBs but I don't trust it. I would either replace the downstairs hub with a switch, or I would do this: You've misunderstood the topology slightly. The server has two fxp interfaces, one is connected to the DSL router, and the other is connected to a hub which is chained to another hub. The router is also an unused hub. It would also be a good idea to test the laptop more directly, by connecting the FXP server to a switch and then connecting the laptop directly to the same switch. I've connected the laptop to the both of the server's fxps, in the first case using the combinations of the upstairs and downstairs hub, and in the later case via the hub on the router on the other side. I get exactly the same behaviour. This to me says that it's not the hubs. To test it further I'll build an ethernet crosser cable and plug the laptop directly into the server. That'll settle it. Joe msg30566/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
:I've connected the laptop to the both of the server's fxps, in the first :case using the combinations of the upstairs and downstairs hub, and in :the later case via the hub on the router on the other side. I get :exactly the same behaviour. This to me says that it's not the hubs. :To test it further I'll build an ethernet crosser cable and plug the :laptop directly into the server. That'll settle it. : :Joe What kind of ethernet does the laptop have? -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:I'm going to blame the USB ethernet driver for dropping packets; Thomas :Zenker has been reporting similar problems on -net. He says that he did :not have problems with packet loss with 4.3, but has not been able to :track down what changed. Maybe you could compare notes with him and see :what you can come up with. : :Mike Silby Silbersack new-reno was artifically limiting the max number of in-transit packets to 4. This is probably why the USB ethernet worked with 4.3, but it destroyed TCP performance for everything else and was removed. I don't think there is kernel solution to the USB problems, short of fixing the driver (if that's even possible). The packets are being sent from the server and the only thing the client has control over is the receive window advertisement. You could artifically reduce the window advertisement on the client to deal with lots of small packets but it would destroy performance for larger packets / data streams. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Dec 28, 2001 at 07:27:36PM +0100, Tomas Svensson wrote:
This just verifies what I said weeks ago.
On the client side:
## several packets are lost here due to congestion, thanks to
TCP_NODELAY:
No. These packets aren't lost to congestion. I can reproduce this
pattern every time. There's no random packet loseness about it.
You are right that switching TCP_NODELAY off does fix it, but it's not
caused by congestion I can assure you.
Joe
Index: packet.c
===
RCS file: /home/ncvs/src/crypto/openssh/packet.c,v
retrieving revision 1.1.1.1.2.4
diff -u -r1.1.1.1.2.4 packet.c
--- packet.c28 Sep 2001 01:33:34 - 1.1.1.1.2.4
+++ packet.c29 Dec 2001 17:33:40 -
@@ -1281,9 +1281,11 @@
error(setsockopt IPTOS_LOWDELAY: %.100s,
strerror(errno));
}
+#if NEVER
if (setsockopt(connection_in, IPPROTO_TCP, TCP_NODELAY, (void *) on,
sizeof(on)) 0)
error(setsockopt TCP_NODELAY: %.100s, strerror(errno));
+#endif
} else if (packet_connection_is_ipv4()) {
/*
* Set IP options for a non-interactive connection. Use
msg30540/pgp0.pgp
Description: PGP signature
Re: FreeBSD performing worse than Linux?
On Sat, 29 Dec 2001, Josef Karthauser wrote: On Fri, Dec 28, 2001 at 07:27:36PM +0100, Tomas Svensson wrote: This just verifies what I said weeks ago. On the client side: ## several packets are lost here due to congestion, thanks to TCP_NODELAY: No. These packets aren't lost to congestion. I can reproduce this pattern every time. There's no random packet loseness about it. You are right that switching TCP_NODELAY off does fix it, but it's not caused by congestion I can assure you. Joe Whether the packet loss is random or not, there is packet loss occuring. What's the exact network setup between the two machines? Perhaps it's a collision occuring each time. I guess you could try running netstat -i on each computer before and after to see if you can see the problem occur at that layer. Mike Silby Silbersack To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sat, Dec 29, 2001 at 12:49:02PM -0500, Mike Silbersack wrote: You are right that switching TCP_NODELAY off does fix it, but it's not caused by congestion I can assure you. Whether the packet loss is random or not, there is packet loss occuring. What's the exact network setup between the two machines? Perhaps it's a collision occuring each time. genius(laptop)/aue0 usb ethernet. (-current) | 10 base half duplex hub | 10 base half duplex hub | transwarp(dual processor server)/fxp0 onboard ethernet. (-stable). I guess you could try running netstat -i on each computer before and after to see if you can see the problem occur at that layer. I wish that I had an ether-crosser cable so I could plug the two machines directly together. I've had this problem for several weeks now. I would imagine that it would be slightly random if it was the network infrastructure, but it's not. Joe msg30542/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
On Sat, 29 Dec 2001, Josef Karthauser wrote: genius(laptop)/aue0 usb ethernet. (-current) | 10 base half duplex hub | 10 base half duplex hub | transwarp(dual processor server)/fxp0 onboard ethernet. (-stable). I'm going to blame the USB ethernet driver for dropping packets; Thomas Zenker has been reporting similar problems on -net. He says that he did not have problems with packet loss with 4.3, but has not been able to track down what changed. Maybe you could compare notes with him and see what you can come up with. Mike Silby Silbersack To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Sat, Dec 29, 2001 at 01:59:28PM -0500, Mike Silbersack wrote: I'm going to blame the USB ethernet driver for dropping packets; Thomas Zenker has been reporting similar problems on -net. He says that he did not have problems with packet loss with 4.3, but has not been able to track down what changed. Maybe you could compare notes with him and see what you can come up with. What's his email address? Joe msg30544/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
Whether the packet loss is random or not, there is packet loss occuring. What's the exact network setup between the two machines? Perhaps it's a collision occuring each time. Please don't imply that normal half duplex Ethernet collisions cause packet loss. They don't. Steinar Haug, Nethelp consulting, [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Saturday, December 29, 2001, 6:41:33 PM, you wrote: JK On Fri, Dec 28, 2001 at 07:27:36PM +0100, Tomas Svensson wrote: ## several packets are lost here due to congestion, thanks to TCP_NODELAY: JK No. These packets aren't lost to congestion. I can reproduce this JK pattern every time. There's no random packet loseness about it. JK You are right that switching TCP_NODELAY off does fix it, but it's not JK caused by congestion I can assure you. I didn't know that this was on a LAN, then you have bigger problems than TCP_NODELAY. I never had problems using OpenSSH on a LAN, but on links with higher latency the burst of small packets does cause congestion and delays in the output (SSH Inc sshd 3.0 does not). -Tomas To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Ok, there's packet loss. From this extract you can see that the client receives through sequence 641, then the next packet it receives starts at sequence 993. 15:28:09.879928 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 609:641(32) ack 64 win 33304 nop,nop,timestamp 152269835 86 8399 (DF) [tos 0x10] 15:28:09.881926 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 993:1025(32) ack 64 win 33304 nop,nop,timestamp 152269835 8 68402 (DF) [tos 0x10] If you look at the server you can see the (tiny) packets being transmitted: 15:28:18.255648 transwarp.telnet genius.kpop: P 609:641(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:18.255775 transwarp.telnet genius.kpop: P 641:673(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:18.255864 genius.kpop transwarp.telnet: . ack 97 win 33288 nop,nop,timestamp 868402 152269835 (DF) [tos 0x10] 15:28:18.255955 transwarp.telnet genius.kpop: P 673:705(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256084 transwarp.telnet genius.kpop: P 705:737(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256223 transwarp.telnet genius.kpop: P 737:769(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256351 transwarp.telnet genius.kpop: P 769:801(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256479 transwarp.telnet genius.kpop: P 801:833(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256607 transwarp.telnet genius.kpop: P 833:865(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256734 transwarp.telnet genius.kpop: P 865:897(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.256884 transwarp.telnet genius.kpop: P 897:929(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.257011 transwarp.telnet genius.kpop: P 929:961(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.257138 transwarp.telnet genius.kpop: P 961:993(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.258068 transwarp.telnet genius.kpop: P 993:1025(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10] 15:28:18.258221 transwarp.telnet genius.kpop: P 1025:1057(32) ack 64 win 33304 nop,nop,timestamp 152269835 868402 (DF) [tos 0x10 The first one is through 641. The server then sends 11 packets (641 through 993) that the client misses completely. The next packet the client sees is the 993:1025 packet. So there is nothing wrong with the TCP protocol. The question now is whether this is packet loss due to the physical layer or whether it is a queueing problem. What kind of network cards do these machines have and what kind of switching infrastructure is between them? One thing that doesn't make sense is that the client is responding to each packet with its own ack but the server doesn't see the acks until it finishes transmitting a big stream of data packets. This implies half-duplex operation. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 03:45:21PM -0800, Matthew Dillon wrote: :... : I am tracking it down now. : :Is this the same problem that I experience on ssh connections between :my 5.0-current laptop and my releng_4 server? When I run an 'ls' :from the shell on large directories I get the response back block :delay block delay block. I assumed that it was a problem with :-current. : :Joe It sounds like the same problem. In fact, I seem to recall observing something very similar from my laptop while ssh'd into one of my servers, but at the time I though it was a hicup in the wireless network. Now though I think it was this same issue. Ok, at last, here's the tcpdumps from both ends. On the client (genius.tao.org.uk -current) I 'ssh -p 23 transwarp' and from there 'ls -l /usr/src'. The tcpdump is from when I hit return on the ls command. The server (transwarp.tao.org.uk) is running -stable. Joe Client side. 15:28:09.819585 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: P 32:64(32) ack 33 win 33304 nop,nop,timestamp 868399 152269076 (DF) [tos 0x10] 15:28:09.841939 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 33:65(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.845937 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 65:97(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.845987 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 97 win 33288 nop,nop,timestamp 868402 152269835 (DF) [tos 0x10] 15:28:09.847937 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 97:129(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.849931 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 129:161(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.849966 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 161 win 33288 nop,nop,timestamp 868402 152269835 (DF) [tos 0x10] 15:28:09.851935 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 161:193(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.853984 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 193:225(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.854032 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 225 win 33288 nop,nop,timestamp 868403 152269835 (DF) [tos 0x10] 15:28:09.855959 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 225:257(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.857934 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 257:289(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.857986 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 289 win 33288 nop,nop,timestamp 868403 152269835 (DF) [tos 0x10] 15:28:09.859937 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 289:321(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.861929 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 321:353(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.861967 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 353 win 33288 nop,nop,timestamp 868403 152269835 (DF) [tos 0x10] 15:28:09.863933 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 353:385(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.865928 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 385:417(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.865963 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 417 win 33288 nop,nop,timestamp 868404 152269835 (DF) [tos 0x10] 15:28:09.867932 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 417:449(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.869928 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 449:481(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.869963 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 481 win 33288 nop,nop,timestamp 868404 152269835 (DF) [tos 0x10] 15:28:09.871932 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 481:513(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.873941 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 513:545(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.873981 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 545 win 33288 nop,nop,timestamp 868405 152269835 (DF) [tos 0x10] 15:28:09.875949 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 545:577(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF) [tos 0x10] 15:28:09.877927 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 577:609(32) ack 64 win 33304 nop,nop,timestamp 152269835 868399 (DF)
Re: FreeBSD performing worse than Linux?
This just verifies what I said weeks ago. On the client side: ## several packets are lost here due to congestion, thanks to TCP_NODELAY: 15:28:09.879928 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 609:641(32) ack 64 win 33304 15:28:09.881926 transwarp.tao.org.uk.telnet genius.tao.org.uk.kpop: P 993:1025(32) ack 64 win 33304 ## client only acks up to 641 after this: 15:28:09.881964 genius.tao.org.uk.kpop transwarp.tao.org.uk.telnet: . ack 641 win 33304 On the server side: 15:28:18.291734 genius.kpop transwarp.telnet: . ack 641 win 33304 ## server resends 641:1057 after a timer expired, 1 second seems long? 15:28:19.288745 transwarp.telnet genius.kpop: P 641:1057(416) ack 64 win 33304 Conclusion: This is a OpenSSH problem, not a FreeBSD problem. -Tomas Friday, December 28, 2001, 4:33:30 PM, you wrote: JK On Fri, Nov 30, 2001 at 03:45:21PM -0800, Matthew Dillon wrote: :... : I am tracking it down now. : :Is this the same problem that I experience on ssh connections between :my 5.0-current laptop and my releng_4 server? When I run an 'ls' :from the shell on large directories I get the response back block :delay block delay block. I assumed that it was a problem with :-current. : :Joe It sounds like the same problem. In fact, I seem to recall observing something very similar from my laptop while ssh'd into one of my servers, but at the time I though it was a hicup in the wireless network. Now though I think it was this same issue. JK Ok, at last, here's the tcpdumps from both ends. JK On the client (genius.tao.org.uk -current) I 'ssh -p 23 transwarp' JK and from there 'ls -l /usr/src'. The tcpdump is from when I hit JK return on the ls command. The server (transwarp.tao.org.uk) is running JK -stable. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Dec 28, 2001 at 07:27:36PM +0100, Tomas Svensson wrote: This just verifies what I said weeks ago. [cut] Conclusion: This is a OpenSSH problem, not a FreeBSD problem. My hunch is that it's not an OpenSSH problem. I also get the same stalling whilst downloading large files in the same direction from the same machines using 'fetch' on the client and apache on the server. Joe msg30494/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
On Fri, Dec 28, 2001 at 10:09:11PM +, Josef Karthauser wrote: On Fri, Dec 28, 2001 at 07:27:36PM +0100, Tomas Svensson wrote: This just verifies what I said weeks ago. [cut] Conclusion: This is a OpenSSH problem, not a FreeBSD problem. My hunch is that it's not an OpenSSH problem. I also get the same stalling whilst downloading large files in the same direction from the same machines using 'fetch' on the client and apache on the server. Here's the detail: genius% fetch http://some.file/on/the/server Receiving book.pdf (7403191 bytes): 100% 7403191 bytes transferred in 36.1 seconds (200.41 kBps) genius% fetch http://some.file/on/the/server Receiving book.pdf (7403191 bytes): 100% 7403191 bytes transferred in 32.0 seconds (226.20 kBps) genius% fetch http://some.file/on/the/server Receiving book.pdf (7403191 bytes): 100% 7403191 bytes transferred in 51.1 seconds (141.62 kBps) genius% fetch http://some.file/on/the/server Receiving book.pdf (7403191 bytes): 100% 7403191 bytes transferred in 37.0 seconds (195.16 kBps) genius% fetch http://some.file/on/the/server Receiving book.pdf (7403191 bytes): 100% 7403191 bytes transferred in 23.2 seconds (311.39 kBps) genius% Joe msg30496/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
On Fri, 28 Dec 2001, Tomas Svensson wrote: This just verifies what I said weeks ago. On the client side: ## several packets are lost here due to congestion, thanks to TCP_NODELAY: ## server resends 641:1057 after a timer expired, 1 second seems long? 15:28:19.288745 transwarp.telnet genius.kpop: P 641:1057(416) ack 64 win 33304 Conclusion: This is a OpenSSH problem, not a FreeBSD problem. -Tomas I'd have to agree. I just recompiled libssh / sshd so that NODELAY would not be set in any case, and ls acts a lot more nicely over a modem link. (2 large updates, rather than lots of tiny ones.) I'm not sure that we're seeing any actual bug in the tcp stack here as much as the effect of packet loss and tinygrams. Mike Silby Silbersack To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Dec 28, 2001 at 05:36:22PM -0500, Mike Silbersack wrote: I'd have to agree. I just recompiled libssh / sshd so that NODELAY would not be set in any case, and ls acts a lot more nicely over a modem link. (2 large updates, rather than lots of tiny ones.) I'm not sure that we're seeing any actual bug in the tcp stack here as much as the effect of packet loss and tinygrams. Packet loss is, or should be, next to zero as both machines are on an uncongested 10base ether on my local network. Joe msg30501/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
On Fri, 28 Dec 2001, Josef Karthauser wrote: On Fri, Dec 28, 2001 at 05:36:22PM -0500, Mike Silbersack wrote: I'd have to agree. I just recompiled libssh / sshd so that NODELAY would not be set in any case, and ls acts a lot more nicely over a modem link. (2 large updates, rather than lots of tiny ones.) I'm not sure that we're seeing any actual bug in the tcp stack here as much as the effect of packet loss and tinygrams. Packet loss is, or should be, next to zero as both machines are on an uncongested 10base ether on my local network. Joe Well, should != is. Look at the packet trace, a couple of packets clearly get dropped. Mike Silby Silbersack To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
Nate Williams [EMAIL PROTECTED] wrote: Unfortunately, I'm unable to run tcpdump on the client, since it's running NT and we're not allowed to install any 3rd party apps on it (such as the WinDump package). NT??? You wouldn't happen to be seeing performance problems with Samba, I hope? There are some known Samba/FreeBSD issues that can cause abysmal performance (~30-40KB/sec -- yes, kilobytes/sec), even with 100BT cards. No hangs or aborts, though. I've only seen this problem with DOS network stacks, but it wouldn't surprise me if it also affected some version of NT. -- Darryl Okahata [EMAIL PROTECTED] DISCLAIMER: this message is the author's personal opinion and does not constitute the support, opinion, or policy of Agilent Technologies, or of the little green men that have been following him all day. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
Unfortunately, I'm unable to run tcpdump on the client, since it's running NT and we're not allowed to install any 3rd party apps on it (such as the WinDump package). NT??? You wouldn't happen to be seeing performance problems with Samba, I hope? We're not using Samba over 100's of miles. :) There are some known Samba/FreeBSD issues that can cause abysmal performance (~30-40KB/sec -- yes, kilobytes/sec), even with 100BT cards. This may be due to problems that Matt Dillon just recently fixed this weekend in FreeBSD's TCP/IP stack. Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
hum, do i get a speed ticket?
i did some tests before applying your patches:
lizard ./tbench 1 dev
.1 clients started
..+*
Throughput 6.10567 MB/sec (NB=7.63209 MB/sec 61.0567 MBit/sec)
lizard ./tbench 2 dev
..2 clients started
++**
Throughput 7.71796 MB/sec (NB=9.64745 MB/sec 77.1796 MBit/sec)
lizard ./tbench 3 dev
...3 clients started
.++.+***
Throughput 9.17012 MB/sec (NB=11.4627 MB/sec 91.7012 MBit/sec)
lizard ./tbench 4 dev
4 clients started
Throughput 10.3365 MB/sec (NB=12.9207 MB/sec 103.365 MBit/sec)
lizard ./tbench 5 dev
.5 clients started
.+...++.+.+
*
Throughput 10.5219 MB/sec (NB=13.1523 MB/sec 105.219 MBit/sec)
Richard (and others), please try this patch. With this patch I
get the following between two machines connected via a 100BaseTX
switch (full duplex):
test1:/home/dillon/dbench ./tbench 1 test2
.1 clients started
..+*
Throughput 6.13925 MB/sec (NB=7.67406 MB/sec 61.3925 MBit/sec) 1 procs
test1:/home/dillon/dbench ./tbench 2 test2
..2 clients started
++**
Throughput 8.37795 MB/sec (NB=10.4724 MB/sec 83.7795 MBit/sec) 2 procs
On localhost I get:
test1:/home/dillon/dbench ./tbench 1 localhost
.1 clients started
..+*
Throughput 25.7156 MB/sec (NB=32.1445 MB/sec 257.156 MBit/sec) 1 procs
test1:/home/dillon/dbench ./tbench 2 localhost
..2 clients started
++**
Throughput 36.5428 MB/sec (NB=45.6785 MB/sec 365.428 MBit/sec) 2 procs
test1:/home/dillon/dbench
This is WITHOUT changing the default send and receive tcp buffers..
they're both 16384.
The bug I found is that when recv() is used with MSG_WAITALL,
which is what tbench does, soreceive() will block waiting for all
available input WITHOUT ever calling pr-pr_usrreqs-pru_rcvd(),
which means that if the sender filled up the receive buffer (16K default)
the receiver will never ack the 0 window... that is until the idle code
takes over after 5 seconds.
-Matt
Index: uipc_socket.c
===
RCS file: /home/ncvs/src/sys/kern/uipc_socket.c,v
retrieving revision 1.68.2.16
diff -u -r1.68.2.16 uipc_socket.c
--- uipc_socket.c 2001/06/14 20:46:06 1.68.2.16
+++ uipc_socket.c 2001/12/01 21:09:13
@@ -910,6 +910,14 @@
!sosendallatonce(so) !nextrecord) {
if (so-so_error || so-so_state SS_CANTRCVMORE)
break;
+ /*
+ * The window might have closed to zero, make
+ * sure we send an ack now that we've drained
+ * the buffer or we might end up blocking until
+ * the idle takes over (5 seconds).
+ */
+ if (pr-pr_flags PR_WANTRCVD so-so_pcb)
+ (*pr-pr_usrreqs-pru_rcvd)(so, flags);
error = sbwait(so-so_rcv);
if (error) {
sbunlock(so-so_rcv);
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-hackers in the body of the message
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-hackers in the body of the message
Re: Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
That's reasonable with patch #2 and delayed acks turned off (assuming this is a 100BaseTX network you are testing on). You should get slightly better results with Patch #3 and delayed acks turned on. -Matt :hum, do i get a speed ticket? :i did some tests before applying your patches: : :lizard ./tbench 1 dev :.1 clients started :..+* :Throughput 6.10567 MB/sec (NB=7.63209 MB/sec 61.0567 MBit/sec) :lizard ./tbench 2 dev :..2 clients started :++** :Throughput 7.71796 MB/sec (NB=9.64745 MB/sec 77.1796 MBit/sec) :lizard ./tbench 3 dev :...3 clients started :.++.+*** :Throughput 9.17012 MB/sec (NB=11.4627 MB/sec 91.7012 MBit/sec) :lizard ./tbench 4 dev :4 clients started : :Throughput 10.3365 MB/sec (NB=12.9207 MB/sec 103.365 MBit/sec) :lizard ./tbench 5 dev :.5 clients started :.+...++.+.+ :* :Throughput 10.5219 MB/sec (NB=13.1523 MB/sec 105.219 MBit/sec) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
That's reasonable with patch #2 and delayed acks turned off (assuming this is a 100BaseTX network you are testing on). You should get slightly better results with Patch #3 and delayed acks turned on. -Matt i did not apply any patches - yet -, net.inet.tcp.newreno is 1, on both hosts. they are connected at 100mb, so the strange numbers are when the number of clients is 3! (att 100Mbit/sec how come i get 100?) danny PS: just forgot, congrat on a terrific job!!! im still amazed :hum, do i get a speed ticket? :i did some tests before applying your patches: : :lizard ./tbench 1 dev :.1 clients started :..+* :Throughput 6.10567 MB/sec (NB=7.63209 MB/sec 61.0567 MBit/sec) :lizard ./tbench 2 dev :..2 clients started :++** :Throughput 7.71796 MB/sec (NB=9.64745 MB/sec 77.1796 MBit/sec) :lizard ./tbench 3 dev :...3 clients started :.++.+*** :Throughput 9.17012 MB/sec (NB=11.4627 MB/sec 91.7012 MBit/sec) :lizard ./tbench 4 dev :4 clients started : :Throughput 10.3365 MB/sec (NB=12.9207 MB/sec 103.365 MBit/sec) :lizard ./tbench 5 dev :.5 clients started :.+...++.+.+ :* :Throughput 10.5219 MB/sec (NB=13.1523 MB/sec 105.219 MBit/sec) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
: slightly better results with Patch #3 and delayed acks turned on. : : -Matt :i did not apply any patches - yet -, net.inet.tcp.newreno is 1, on both hosts. :they are connected at 100mb, so the strange numbers are when the number of :clients is 3! (att 100Mbit/sec how come i get 100?) It's a synchronous protocol, so the number of bits per byte is not necessarily 10... it can be 8, 9, or 10. Approximately. : danny :PS: just forgot, congrat on a terrific job!!! im still amazed -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Hi, On Sat, Dec 01, 2001 at 02:07:41PM +1030, Greg Lehey wrote: so my drive is now running at UDMA 100. Can you send me dmesg output? In particular, I had a printf output there to show what the BIOS had set. Background for other people: Richard has an IDE chip which claims to be a SiS 5591, which according to the data sheet can't do better than UDMA 33. When he runs Linux on the box, however, it claims to be running at UDMA 100, and this hack seems to have had the same effect. Is this patch available somewhere, or could you send it to me? I have the same chipset on an Asus Athlon mobo: atapci0: SiS 5591 ATA33 controller port 0xd800-0xd80f at device 0.1 on pci0 And yes, Linux can run it at UDMA100, and the printed datasheet of the board says it's an UDMA100-capable chip. (The SiS doc says UDMA33). Thanks __ o _. __ / / / (_(_(__(_) @ bsd.hu To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
Matthew Dillon wrote: Index: tcp_output.c === RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v retrieving revision 1.39.2.10 diff -u -r1.39.2.10 tcp_output.c --- tcp_output.c 2001/07/07 04:30:38 1.39.2.10 +++ tcp_output.c 2001/11/30 21:18:10 @@ -912,7 +912,14 @@ tp-t_flags = ~TF_ACKNOW; if (tcp_delack_enabled) callout_stop(tp-tt_delack); +#if 0 + /* + * This completely breaks TCP if newreno is turned on + */ if (sendalot (!tcp_do_newreno || --maxburst)) + goto again; +#endif + if (sendalot) goto again; return (0); } To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message OK, I have applied this patch, and FreeBSD 4.4-STABLE now seems to behave approximately the same as Linux. There are no extra ACKs, and FreeBSD now coalesces pairs of ACKs. However, performance for one client is still at 25Mb/s with the tbench run, while Linux provides around 68Mb/s. So, it is back to staring at traces. Perhaps I will get a full trace now. -- Richard Sharpe, [EMAIL PROTECTED], LPIC-1 www.samba.org, www.ethereal.com, SAMS Teach Yourself Samba in 24 Hours, Special Edition, Using Samba To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
!@#%@#^%^$#^@#$%#@!$%!@%@#@# Whos the blasted bozo that broke TCP! :-) :-( I believe I have found another problem. It may not be related to this one but I sure hope it is. When the receiver's window contracts to zero causing the sender to stop transmitting, TCP stalls for 5 seconds. In otherwords, some bozo broke the window restart code when the receiver drains its buffer NO ACK IS SENT INDICATING THAT THE WINDOW HAS OPENED UP It's only luck that the TCP idle code picks it up and acks it after 5 seconds. Somebody broke something horrendously in the TCP stack. !$#%@#$^%#@^@#%!@#$@!#$@#$@#$ blast it! I'm going to fry the responsible party when I get this tracked down. -Matt test1# tail x 12:44:25.629324 216.240.41.12.7003 216.240.41.11.1056: . ack 10417344 win 16060 (DF) 12:44:25.629344 216.240.41.11.1056 216.240.41.12.7003: P 10430484:10431944(1460) ack 47215 win 17520 (DF) 12:44:25.629374 216.240.41.11.1056 216.240.41.12.7003: . 10431944:10433404(1460) ack 47215 win 17520 (DF) 12:44:25.629566 216.240.41.12.7003 216.240.41.11.1056: . ack 10420264 win 14600 (DF) 12:44:25.629587 216.240.41.11.1056 216.240.41.12.7003: P 10433404:10434864(1460) ack 47215 win 17520 (DF) 12:44:25.629810 216.240.41.12.7003 216.240.41.11.1056: . ack 10423184 win 11680 (DF) 12:44:25.630056 216.240.41.12.7003 216.240.41.11.1056: . ack 10426104 win 8760 (DF) 12:44:25.630302 216.240.41.12.7003 216.240.41.11.1056: . ack 10429024 win 5840 (DF) 12:44:25.630548 216.240.41.12.7003 216.240.41.11.1056: . ack 10431944 win 2920 (DF) 12:44:25.630794 216.240.41.12.7003 216.240.41.11.1056: . ack 10434864 win 0 (DF) (stall) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
Richard (and others), please try this patch. With this patch I
get the following between two machines connected via a 100BaseTX
switch (full duplex):
test1:/home/dillon/dbench ./tbench 1 test2
.1 clients started
..+*
Throughput 6.13925 MB/sec (NB=7.67406 MB/sec 61.3925 MBit/sec) 1 procs
test1:/home/dillon/dbench ./tbench 2 test2
..2 clients started
++**
Throughput 8.37795 MB/sec (NB=10.4724 MB/sec 83.7795 MBit/sec) 2 procs
On localhost I get:
test1:/home/dillon/dbench ./tbench 1 localhost
.1 clients started
..+*
Throughput 25.7156 MB/sec (NB=32.1445 MB/sec 257.156 MBit/sec) 1 procs
test1:/home/dillon/dbench ./tbench 2 localhost
..2 clients started
++**
Throughput 36.5428 MB/sec (NB=45.6785 MB/sec 365.428 MBit/sec) 2 procs
test1:/home/dillon/dbench
This is WITHOUT changing the default send and receive tcp buffers..
they're both 16384.
The bug I found is that when recv() is used with MSG_WAITALL,
which is what tbench does, soreceive() will block waiting for all
available input WITHOUT ever calling pr-pr_usrreqs-pru_rcvd(),
which means that if the sender filled up the receive buffer (16K default)
the receiver will never ack the 0 window... that is until the idle code
takes over after 5 seconds.
-Matt
Index: uipc_socket.c
===
RCS file: /home/ncvs/src/sys/kern/uipc_socket.c,v
retrieving revision 1.68.2.16
diff -u -r1.68.2.16 uipc_socket.c
--- uipc_socket.c 2001/06/14 20:46:06 1.68.2.16
+++ uipc_socket.c 2001/12/01 21:09:13
@@ -910,6 +910,14 @@
!sosendallatonce(so) !nextrecord) {
if (so-so_error || so-so_state SS_CANTRCVMORE)
break;
+ /*
+* The window might have closed to zero, make
+* sure we send an ack now that we've drained
+* the buffer or we might end up blocking until
+* the idle takes over (5 seconds).
+*/
+ if (pr-pr_flags PR_WANTRCVD so-so_pcb)
+ (*pr-pr_usrreqs-pru_rcvd)(so, flags);
error = sbwait(so-so_rcv);
if (error) {
sbunlock(so-so_rcv);
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-hackers in the body of the message
Re: Patch #2 (was Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?))
Oh, I forgot to mention... there's still a bug. You have to turn off delayed acks to get the below numbers. Otherwise the window 0 recovery ack ... gets delayed. Joy. It's easy to fix, but I'm off to see Potter so it will be tonight. -Matt :Richard (and others), please try this patch. With this patch I :get the following between two machines connected via a 100BaseTX :switch (full duplex): : : : :test1:/home/dillon/dbench ./tbench 1 test2 :.1 clients started :..+* :Throughput 6.13925 MB/sec (NB=7.67406 MB/sec 61.3925 MBit/sec) 1 procs :test1:/home/dillon/dbench ./tbench 2 test2 :..2 clients started :++** :Throughput 8.37795 MB/sec (NB=10.4724 MB/sec 83.7795 MBit/sec) 2 procs : : : : On localhost I get: : : : :test1:/home/dillon/dbench ./tbench 1 localhost :.1 clients started :..+* :Throughput 25.7156 MB/sec (NB=32.1445 MB/sec 257.156 MBit/sec) 1 procs :test1:/home/dillon/dbench ./tbench 2 localhost :..2 clients started :++** :Throughput 36.5428 MB/sec (NB=45.6785 MB/sec 365.428 MBit/sec) 2 procs :test1:/home/dillon/dbench : : : :This is WITHOUT changing the default send and receive tcp buffers.. :they're both 16384. : :The bug I found is that when recv() is used with MSG_WAITALL, :which is what tbench does, soreceive() will block waiting for all :available input WITHOUT ever calling pr-pr_usrreqs-pru_rcvd(), :which means that if the sender filled up the receive buffer (16K default) :the receiver will never ack the 0 window... that is until the idle code :takes over after 5 seconds. : : -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 10:28:09PM -0500, Leo Bicknell wrote: I can't reproduce this result, 16K fills a T1 for 11 ms, which is 22000 km (at 2/3 of light speed), enough to get halfway round the Your math is a little funny. Right, I knew there was something wrong somewhere :-) 4000 km one way == 8000 km two way, 8000 / 168300 = 47ms in my book, theoretial optimum. With an RTT of 47ms, you can move 16k per RTT, or or about 340k/sec. It's where I don't quite agree: for a bulk transfer, there is no RTT to account for, you only need to take into account the one-way delay, TCP does the rest for you assuming the window is large enough. Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:I managed to track the problem down to the duplex settings on both the :Ethernet cards (AT-2500 TX, Realtek 8139 based, AFAIK) and the 10/100 :Switch. Forcing both the cards and the switch to particular settings :cured the problem, and lead to a massive performance increase. : :FTP seems to be particularly badly affected by the constant collisions :(causing backoff). The problem can be tricky to find as the switch :wasn't perceptably showing collisions on the collision LED, but viewing :the switch stats showed a different story! It probably wasn't collision backoff. It was probably packet loss due to the switch and the host getting confused over the duplex setting. i.e. where the switch thought it was one thing and the host thought it was another. The usual solution to this sort of switch/host confusion is to turn off autonegotiation on both sides and hardwire the duplex to full. On both sides. I've occassionally had similar problems (not in the last year or two, though... but definitely with older cards and switches). -Matt :I've noticed similar problems with Linux and certain cards (it was a :while ago). : :John Vinters :[EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
: FWIW, I'm seeing this as well. However, this appears to be a new : occurance, as we were using a FreeBSD 3.X system for our reference test : platform. : :Someone recently submitted a PR about TCP based NFS being significantly :slower under 4.X. I wonder if it could be related? : : http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/32141 : :There is quite a lot of detail in the PR and the submitter has no :trouble reproducing the problem. : : David. Hmm. I'll play with it a bit tomorrow. Er, later today. One thing I noticed recently with NFS/TCP is that I have to run 'nfsiod -n 4' on the client to get reasonable TCP performance. I don't think I had to do that before. It sure sounds similar... like a delayed-ack problem or improper transmit side backoff. It would be nice if someone able to reproduce the problem can test the TCP connection with newreno turned off (net.inet.tcp.newreno) and delayed acks turned off (net.inet.tcp.delayed_ack). If that fixes the proble it narrows down our search considerably. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 11:11:56AM +0100, Pierre Beyssac wrote: On Thu, Nov 29, 2001 at 10:28:09PM -0500, Leo Bicknell wrote: 4000 km one way == 8000 km two way, 8000 / 168300 = 47ms in my book, theoretial optimum. With an RTT of 47ms, you can move 16k per RTT, or or about 340k/sec. It's where I don't quite agree: for a bulk transfer, there is no RTT to account for, you only need to take into account the one-way delay, TCP does the rest for you assuming the window is large enough. Asume you have 10ms one way delay and an RTT of 20ms. Lets asume your windowsize fits exactly the one way delay. You start sending data until the send window is exhaustet. You have been seending 10ms from the begining and at that time the first packet of you stream reaches the receiver. Now you have to stop sending data for 10ms because you have to wait for the first acknowledge to arive to free some space of the window - if the receiver delays ack you have to wait longer. You can easily see that you need to take RTT + delayed-ack-time into acount. -- B.Walter COSMO-Project http://www.cosmo-project.de [EMAIL PROTECTED] Usergroup [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Quoting Bruce A. Mah ([EMAIL PROTECTED]): How early in November? I'm staring at this commit message and wondering if it has any relevance to your situation: - revision 1.107.2.18 date: 2001/11/12 22:11:24; author: nate; state: Exp; lines: +3 -1 MFH: V1.139 when newreno is turned on, if dupacks = 1 or dupacks = 2 and new data is acknowledged, reset the dupacks to 0. The problem was spotted when a connection had its send buffer full because the congestion window was only 1 MSS and was not being incremented because dupacks was not reset to 0. Reviewed by:jlemon - Kernel was built on November 7 probably from sources a day or two earlier. The source tree has been updated since the build. Nate's commit above is not in my kernel. For those interested the server and client dumps are here: http://www.irbs.net/server-dump.html http://www.irbs.net/client-dump.html The server clock looks like its about 900Ms ahead of the client. John Capo To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
(snip...a large number of postings regarding slow performance by 4.x kernels with TCP/IP) A friend who works for a local university and I tried moving large files using variouis OS'es and hardware. These are FTP transfers with file sizes from 100 to 300 megabytes.. The conclusion we arrived at was that the TCP performance of FreeBSD 4.x and Linux is aproximately the same and that processor speed makes the most difference. In one case, a fast laptop with 16 bit pcmcia NIC did poorly. Moving large files on 100mb/s ethernet backbones gave the folowing results... Dual 800 mhz PIII processors with Linux 6.1: 10mB/s. Sunblade 100's: 10mB/s. Single 1.4ghz processors (noname box)with 3C905 NICS, FreeBSD-stable (June 2001).: 9.5 mB/s. In the case wehere we had only one machine of a type, we used the dual 800mhz machines as a sink...with the following results (this is probably questionable): Dual 333 Linux 5.1 5mB/s Pentium 350 III with 3C905 NIC, Linux 5.1: 2mB/sec K6-2 400 with smc NIC, Linux 5.1: 2.8mB/sec Dell 500mhz PowerEdge with 4.3 with 3C905 NIC to HP Netserver PII 266, both running 4.3-RELEASE: 3.0 mB/sec. Dell 500mhz PowerEdge with 4.3 to Dell 850mhz laptop running 4.4 with Dlink PCMCIA ethernet card: 1.0 mB/sec. (caused by pcmcia NIC?) PIIMMX 200mhz box running 4.4-Relese with 3C905 to same Dell Laptop: 500kB/sec. Unfortunately, we didn't have any 7.x Linux available or 3.X FreeBSD. FWIW... Jim Durham To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 10:31:56PM -0500, Jonathan M. Slivko wrote: If you give me your IP address, I can ping *from* Columbia.edu to your machine and see what I get, that should pretty much solve any issues that may arise. pun.isi.edu 128.9.160.150 Thanks. msg29374/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
: FWIW, I'm seeing this as well. However, this appears to be a new : occurance, as we were using a FreeBSD 3.X system for our reference test : platform. : :Someone recently submitted a PR about TCP based NFS being significantly :slower under 4.X. I wonder if it could be related? : : http://www.freebsd.org/cgi/query-pr.cgi?pr=misc/32141 : :There is quite a lot of detail in the PR and the submitter has no :trouble reproducing the problem. : : David. Hmm. I'll play with it a bit tomorrow. Er, later today. One thing I noticed recently with NFS/TCP is that I have to run 'nfsiod -n 4' on the client to get reasonable TCP performance. I don't think I had to do that before. It sure sounds similar... like a delayed-ack problem or improper transmit side backoff. It would be nice if someone able to reproduce the problem can test the TCP connection with newreno turned off (net.inet.tcp.newreno) and delayed acks turned off (net.inet.tcp.delayed_ack). If that fixes the proble it narrows down our search considerably. John Capo replied that turning off both did not help his setup any. I was supposed to be testing things yesterday, but the guys got pulled away on another project. Perhaps today I'll get a chance to get some tcpdump's and some more test data. Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Looking at the complete dump on the server more closely I see what's happening. The server didn't jump ahead in the stream. The client side of these tests is on a fractional T1. In about 60Ms the server pushed a window's worth of data, about 200 packets since the payload was small, 48 bytes. (48 + IP + TCP) * 200 is around 17KB in 60Ms which overflowed the frame switch queue. The other part of the dump where the server is acked for a segment just sent but does not send the next segment till a duplicate ack is received better than a second later is still suspect to me. John Capo Quoting Sergey Babkin ([EMAIL PROTECTED]): And here a _very_ pathological thing has happened: the server just forgot to send the data between sequence numbers 12937 and 28049. Since the dump was done on the server side, this suggests that something very bad has happened with the TCP state on the server side. Possibly the value of the current sequence number in the protocol control block got overwritten by something. To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
In article local.mail.freebsd-hackers/[EMAIL PROTECTED] you write: Quoting Sergey Babkin ([EMAIL PROTECTED]): John Capo wrote: 21:41:49.001039 client.4427 server.22: P 144:192(48) ack 12937 win 17376 nop,nop,timestamp 53827954 105528895 (DF) [tos 0x10] 21:41:49.001073 server.22 client.4427: . 28049:29497(1448) ack 192 win 17328 nop,nop,timestamp 105529049 53827954 (DF) [tos 0x10] 21:41:49.001085 server.22 client.4427: P 29497:30313(816) ack 192 win 17328 nop,nop,timestamp 105529049 53827954 (DF) [tos 0x10] 21:41:49.109131 client.4427 server.22: . ack 12937 win 17376 nop,nop,timestamp 53827967 105528895 (DF) [tos 0x10] And here a _very_ pathological thing has happened: the server just forgot to send the data between sequence numbers 12937 and 28049. Since the dump was done on the server side, this suggests that something very bad has happened with the TCP state on the server side. Possibly the value of the current sequence number in the protocol control block got overwritten by something. I don't believe this is happening. It looks like the server blasts everything over the the client, and the client drops a whole bunch of segments. When the server gets the dupack, it correctly performs a fast retransmit and continues transmitting where it left off. server side: 21:41:46.396051 client.4427 server.22: . ack 11489 win 17376 21:41:46.418208 client.4427 server.22: . ack 11489 win 17376 21:41:47.460903 server.22 client.4427: . 11489:12937(1448) ack 144 win 17376 client side: 21:41:46.712307 server.22 client.4427: P 11441:11489(48) ack 144 win 17376 21:41:46.763034 server.22 client.4427: . 25937:27385(1448) ack 144 win 17376 21:41:46.763106 client.4427 server.22: . ack 11489 win 17376 21:41:46.785324 server.22 client.4427: P 27385:28049(664) ack 144 win 17376 21:41:46.785370 client.4427 server.22: . ack 11489 win 17376 21:41:47.936278 server.22 client.4427: . 11489:12937(1448) ack 144 win 17376 However, at this point, the client no further packets, so the server really needs to enter slow start and retransmit everything starting at 12937. Instead, it seems that the server remains in congestion avoidance, and keeps sending at leading edge of the window, performing fast retransmits. John, please try tweaking this sysctl: sysctl -w net.inet.tcp.local_slowstart_flightsize=1 which should force the server to start out doing slow start. This isn't exactly a fix for the above problem, but may heip avoid getting into the situation in the first place. -- Jonathan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Well, this is embarassing. I can reproduce this completely running 4.4-stable (Nov 17th kernel) on two machines. With newreno turned on, a TCP NFS mount only gets 80K/sec. With newreno turned off on the transmit side, a TCP NFS mount gets 7MB/sec. The state of the delayed-ack sysctl is irrelevant. This is without running any nfsiod's (which would mask the degredation of the synchronous messaging). I am tracking it down now. -Matt Matthew Dillon [EMAIL PROTECTED] : (I wrote) :Hmm. I'll play with it a bit tomorrow. Er, later today. One thing :I noticed recently with NFS/TCP is that I have to run 'nfsiod -n 4' :on the client to get reasonable TCP performance. I don't think I :had to do that before. It sure sounds similar... like a delayed-ack :problem or improper transmit side backoff. : :It would be nice if someone able to reproduce the problem can test :the TCP connection with newreno turned off (net.inet.tcp.newreno) :and delayed acks turned off (net.inet.tcp.delayed_ack). If that :fixes the proble it narrows down our search considerably. : :Hello, I am the submitter of PR 32141 mentioned above, : :I did check it with a 4.3 Release server and 4.2 Release client using :'mount_nfs -3 -T ...': :setting net.inet.tcp.newreno=0 gives fast performance (about 8 Mbyte/s, same :for udp mount), setting net.inet.tcp.newreno=1 gives 80kbyte/s. :Setting net.inet.tcp.delayed_ack=0 has no influence (I checked all 4 :combinations). :There is 'nfsiod -n 4' running at clientside (default setting if enabling :nfs via sysinstall). We did not play around with nfsiod settings so far. : :Hope that helps : : Alexander : :-- :Alexander Haderer Charite To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
I believe I have found the problem. The transmit side has a maximum burst count imposed by newreno. As far as I can tell, if this maxburst is hit (it defaults to 4 packets), the transmitter just stops - presumably until it receives an ack. Now, theoretically this should work just fine... send four packets, receive the first ack and send the next four packets... it should allow us to fill the window geometrically. I believe the idea is to give transmit packets a chance to include acks for received data in a reasonable period of time... I'm not sure, it's J Lemon's commit (from the original newreno commits) so maybe he can work it out. However, if the receiver has delayed-acks turned on only one ack is returned for all four packets. The next four are then sent and one ack is returned. I believe this the cause of the problem. It effectively destroys the TCP window, forcing it to around 1.5Kx4 = 6K. This also explains why performance is so weird... if more then one delayed ack happens to occur per burst you get 'bumps' in the performance. Without the patch, two things will solve or partially solve the problem: * Turn off delayed acks on the receiver (performance 80K-6.8MB/sec) OR * Turn off newreno on the transmitter. (performance 80K-7.9MB/sec) The patch below kills the burst limit on the transmit side and appears to solve the problem permanently. I'm sure I'm breaking something in the newreno RFC, but I am going to commit it to both branches now because our current situation is horrible. -Matt Index: tcp_output.c === RCS file: /home/ncvs/src/sys/netinet/tcp_output.c,v retrieving revision 1.39.2.10 diff -u -r1.39.2.10 tcp_output.c --- tcp_output.c2001/07/07 04:30:38 1.39.2.10 +++ tcp_output.c2001/11/30 21:18:10 @@ -912,7 +912,14 @@ tp-t_flags = ~TF_ACKNOW; if (tcp_delack_enabled) callout_stop(tp-tt_delack); +#if 0 + /* +* This completely breaks TCP if newreno is turned on +*/ if (sendalot (!tcp_do_newreno || --maxburst)) + goto again; +#endif + if (sendalot) goto again; return (0); } To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Wierd, on my Dual PII 300 I'm getting around 8MB/sec to an 800MHz athlon. The athlon is using a 3com 905b I believe, and the PII is using an intel fxp type card. Granted this is from my living room to my bedroom so that may be part of what I see. Also, the Dual PII is running -STABLE as of a week ago, and the Athlon is running -CURRENT as of about a week ago. Ken On Fri, 30 Nov 2001, James C. Durham wrote: (snip...a large number of postings regarding slow performance by 4.x kernels with TCP/IP) A friend who works for a local university and I tried moving large files using variouis OS'es and hardware. These are FTP transfers with file sizes from 100 to 300 megabytes.. The conclusion we arrived at was that the TCP performance of FreeBSD 4.x and Linux is aproximately the same and that processor speed makes the most difference. In one case, a fast laptop with 16 bit pcmcia NIC did poorly. Moving large files on 100mb/s ethernet backbones gave the folowing results... Dual 800 mhz PIII processors with Linux 6.1: 10mB/s. Sunblade 100's: 10mB/s. Single 1.4ghz processors (noname box)with 3C905 NICS, FreeBSD-stable (June 2001).: 9.5 mB/s. In the case wehere we had only one machine of a type, we used the dual 800mhz machines as a sink...with the following results (this is probably questionable): Dual 333 Linux 5.1 5mB/s Pentium 350 III with 3C905 NIC, Linux 5.1: 2mB/sec K6-2 400 with smc NIC, Linux 5.1: 2.8mB/sec Dell 500mhz PowerEdge with 4.3 with 3C905 NIC to HP Netserver PII 266, both running 4.3-RELEASE: 3.0 mB/sec. Dell 500mhz PowerEdge with 4.3 to Dell 850mhz laptop running 4.4 with Dlink PCMCIA ethernet card: 1.0 mB/sec. (caused by pcmcia NIC?) PIIMMX 200mhz box running 4.4-Relese with 3C905 to same Dell Laptop: 500kB/sec. Unfortunately, we didn't have any 7.x Linux available or 3.X FreeBSD. FWIW... Jim Durham To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
I believe I have found the problem. The transmit side has a maximum burst count imposed by newreno. As far as I can tell, if this maxburst is hit (it defaults to 4 packets), the transmitter just stops - presumably until it receives an ack. Note, my experiences (and John Capos) are showing degraded performance when *NOT* on a LAN segment. In other words, when packet loss enters the mix, performance tends to fall off rather quickly. This is with or without newreno (which should theoretically help with packet loss). John claims that disabling delayed_ack doesn't seem to affect his performance, and I've not been able to verify if delayed_ack helps/hurts in my situation, since the testers have been pressed for time so I can't get them to iterate through the different settings. I do however have some packet dumps, although I'm not sure they will tell anything. :( Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Alfred Perlstein wrote: * Richard Sharpe [EMAIL PROTECTED] [011130 15:02] wrote: The traffic in the tbench case is SMB taffic. Request/response, with a mixture of small requests and responses, and big request/small response or small request/big response, where big is 64K. I have switched off newreno, and it made no difference. I have switched off delayed_ack, and it reduced performance about 5 percent. I have made sure that SO_SNDBUF and SO_RCVBUF were set to 131072 (which seems to be the max), and it increased performance marginally (like about 2%), but consistently. I am still analysing the packet traces I have, but it seems to me that the crucial difference is Linux seems to delay longer before sending ACKs, and thus sends less ACKs. Since the ACK is piggybacked in the response (or the next request), it all works fine, and the reponse/request gets there sooner. However, I have not convinced myself that the saving of 20uS or so per request/response pair accounts for some 40+ Mb/s. Can you try these two commands: sysctl -w net.inet.tcp.recvspace=65536 sysctl -w net.inet.tcp.sendspace=65536 Yes, that is what I did ... -- Richard Sharpe, [EMAIL PROTECTED], LPIC-1 www.samba.org, www.ethereal.com, SAMS Teach Yourself Samba in 24 Hours, Special Edition, Using Samba To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
:Note, my experiences (and John Capos) are showing degraded performance :when *NOT* on a LAN segment. In other words, when packet loss enters :the mix, performance tends to fall off rather quickly. : :This is with or without newreno (which should theoretically help with :packet loss). John claims that disabling delayed_ack doesn't seem to :affect his performance, and I've not been able to verify if delayed_ack :helps/hurts in my situation, since the testers have been pressed for :time so I can't get them to iterate through the different settings. : :I do however have some packet dumps, although I'm not sure they will :tell anything. :( : :Nate Packet loss will screw up TCP performance no matter what you do. NewReno, assuming it is working properly, can improve performance for that case but it will not completely solve the problem (nothing will). Remember that our timers are only good to around 20ms by default, so even the best retransmission case is going to create a serious hicup. The question here is... is it actually packet loss that is creating this issue for you and John, or is it something else? The only way to tell for sure is to run tcpdump on BOTH the client and server and then observe whether packet loss is occuring by comparing the dumps. I would guess that turning off delayed-acks will improve performance in the face of packet loss, since a lost ack packet in that case will not be as big an issue. -Matt To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
:Note, my experiences (and John Capos) are showing degraded performance :when *NOT* on a LAN segment. In other words, when packet loss enters :the mix, performance tends to fall off rather quickly. : :This is with or without newreno (which should theoretically help with :packet loss). John claims that disabling delayed_ack doesn't seem to :affect his performance, and I've not been able to verify if delayed_ack :helps/hurts in my situation, since the testers have been pressed for :time so I can't get them to iterate through the different settings. : :I do however have some packet dumps, although I'm not sure they will :tell anything. :( : :Nate Packet loss will screw up TCP performance no matter what you do. I know, dealing with that issue is my day job. :) My point is that older FreeBSD releases (and newer Linux releases) seem to be dealing with it in a more sane manner. At least, it didn't effect performance nearly as much as it does in newer releases. NewReno, assuming it is working properly, can improve performance for that case but it will not completely solve the problem (nothing will). Remember that our timers are only good to around 20ms by default, so even the best retransmission case is going to create a serious hicup. See above. The question here is... is it actually packet loss that is creating this issue for you and John, or is it something else? In my opinion, it's how the TCP stack recovers from packet loss that is the problem. The only way to tell for sure is to run tcpdump on BOTH the client and server and then observe whether packet loss is occuring by comparing the dumps. Unfortunately, I'm unable to run tcpdump on the client, since it's running NT and we're not allowed to install any 3rd party apps on it (such as the WinDump package). I'm not saying that I expect the same results as I get on the LAN segment, but I *am* expecting results that are equivalent to what we were seeing with FreeBSD 3.x, and those that are in the same ballpark (or better) than the Linux systems sitting next to it. Given that I get great LAN resuls, I no longer suspect I have a ethernet autonegotiation problem, since I can get almost wire-speeds with local nodes, and close to maximum performance with our wireless products when the network segment the FreeBSD server is relatively idle. I would guess that turning off delayed-acks will improve performance in the face of packet loss, since a lost ack packet in that case will not be as big an issue. I'm not sure I agree. I wouldn't expect it would help/hinder the performance assuming a correctly performing stack, *UNLESS* the packet loss was completely due to congestion. In that case, delayed-acks *may* improve things, but I doubt it would help much with TCP backoff and such. Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
As a side note, I turned off delayed ack on both machines, and had the sendsize and recvsize set at 32768... I'm talking about wirespeed too, not measured incredibly accurately, but just measured using one of the windowmaker dockapps :-D Ken On Fri, 30 Nov 2001, Kenneth Wayne Culver wrote: Wierd, on my Dual PII 300 I'm getting around 8MB/sec to an 800MHz athlon. The athlon is using a 3com 905b I believe, and the PII is using an intel fxp type card. Granted this is from my living room to my bedroom so that may be part of what I see. Also, the Dual PII is running -STABLE as of a week ago, and the Athlon is running -CURRENT as of about a week ago. Ken On Fri, 30 Nov 2001, James C. Durham wrote: (snip...a large number of postings regarding slow performance by 4.x kernels with TCP/IP) A friend who works for a local university and I tried moving large files using variouis OS'es and hardware. These are FTP transfers with file sizes from 100 to 300 megabytes.. The conclusion we arrived at was that the TCP performance of FreeBSD 4.x and Linux is aproximately the same and that processor speed makes the most difference. In one case, a fast laptop with 16 bit pcmcia NIC did poorly. Moving large files on 100mb/s ethernet backbones gave the folowing results... Dual 800 mhz PIII processors with Linux 6.1: 10mB/s. Sunblade 100's: 10mB/s. Single 1.4ghz processors (noname box)with 3C905 NICS, FreeBSD-stable (June 2001).: 9.5 mB/s. In the case wehere we had only one machine of a type, we used the dual 800mhz machines as a sink...with the following results (this is probably questionable): Dual 333 Linux 5.1 5mB/s Pentium 350 III with 3C905 NIC, Linux 5.1: 2mB/sec K6-2 400 with smc NIC, Linux 5.1: 2.8mB/sec Dell 500mhz PowerEdge with 4.3 with 3C905 NIC to HP Netserver PII 266, both running 4.3-RELEASE: 3.0 mB/sec. Dell 500mhz PowerEdge with 4.3 to Dell 850mhz laptop running 4.4 with Dlink PCMCIA ethernet card: 1.0 mB/sec. (caused by pcmcia NIC?) PIIMMX 200mhz box running 4.4-Relese with 3C905 to same Dell Laptop: 500kB/sec. Unfortunately, we didn't have any 7.x Linux available or 3.X FreeBSD. FWIW... Jim Durham To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
Hi, I think that there are two different problems here. My situation involves a LAN (actually, a crossover cable). I have captured a trace of a 1 client run between the Linux driver and the FreeBSD test system as well as between the Linux driver and the same test system running Linux. I am noticing some interesting things. Linux uses the timestamp option in all the TCP segments I have looked at, so it is sending 12 more bytes per segment that FreeBSD. However, more interesting is that for small messages (less that 1460), FreeBSD does not seem to delay sending ACKs, so we get the following pattern: FREEBSD Driver - Test system: 94 byte IP DG with simulated command Test System - Driver: Ack after 83uS Test System - Driver: Psh Ack after 29uS with 79 total bytes in IP DG LINUX Driver - Test system: 106 byte IP DG with simulated command Test System - Driver: Psh Ack after 89uS with 91 total bytes in IP DG So, as you can see, Linux seems to shave some time off each transaction by avoiding sending extra ACKs. Also, what I am seeing is that neither FreeBSD nor Linux is doing ACK coalescing (if that is possible). While I understand that coalescing ACKs will mess up RTT calculations and SRTT a bit, it would serve to reduce the time taken until responses come back. What I am seeing for large transmits is the following: FreeBSD (Test) Linux (Driver) Request, 1500 bytes including request and some data More segments from the request Some ACKs - About one every two segments Last data segment, usually less that 1500 Lots of ACKs one per segment Usually with large window (ie 16020 when the max window seems to be 16384). Response Less than 1500 Now, I have seen something like 10+ ACKS after the driver has finished sending. They appear to be one per sent segment. Then the FreeBSD system sends its response. The optimal would be for the FreeBSD system to delay the ack until it has data to send, which it probably already has. What I see with the Linux trace is that Linux coalesces ACKs. However, the most I have seen it coalesce is two segments. HTH. -- Richard Sharpe, [EMAIL PROTECTED], LPIC-1 www.samba.org, www.ethereal.com, SAMS Teach Yourself Samba in 24 Hours, Special Edition, Using Samba To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
On Fri, Nov 30, 2001 at 04:28:32PM -0600, Alfred Perlstein wrote: * Matthew Dillon [EMAIL PROTECTED] [011130 16:02] wrote: Packet loss will screw up TCP performance no matter what you do. NewReno, assuming it is working properly, can improve performance for that case but it will not completely solve the problem (nothing will). Remember that our timers are only good to around 20ms by default, so even the best retransmission case is going to create a serious hicup. The question here is... is it actually packet loss that is creating this issue for you and John, or is it something else? The only way to tell for sure is to run tcpdump on BOTH the client and server and then observe whether packet loss is occuring by comparing the dumps. I would guess that turning off delayed-acks will improve performance in the face of packet loss, since a lost ack packet in that case will not be as big an issue. I have an odd theory that makes use of my waning remeberence of the stack behavior, this may be totally off base but I'd appreciate it if you guys would consider this scenerio if at all to put my mind at ease. I seem to remeber several places in the stack that detect what looks like a hiccup and immediately begin sending a sequence of ACKs in order to trigger the other side's fast retrasmit code. One of the things that I don't remember seeing is that state is persistant. There isn't anything in the receiver side that does this; ACKs are sent in response to incoming packets. However, state is maintained on the sender side as to whether we are performing fast retransmit or not. -- Jonathan To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
* Jonathan Lemon [EMAIL PROTECTED] [011130 17:00] wrote: On Fri, Nov 30, 2001 at 04:28:32PM -0600, Alfred Perlstein wrote: I have an odd theory that makes use of my waning remeberence of the stack behavior, this may be totally off base but I'd appreciate it if you guys would consider this scenerio if at all to put my mind at ease. I seem to remeber several places in the stack that detect what looks like a hiccup and immediately begin sending a sequence of ACKs in order to trigger the other side's fast retrasmit code. One of the things that I don't remember seeing is that state is persistant. There isn't anything in the receiver side that does this; ACKs are sent in response to incoming packets. However, state is maintained on the sender side as to whether we are performing fast retransmit or not. Either you don't follow or my concept of what happens is off. What i'm saying is this, consider each pair to be in some form of time: h1 send: p1 p2 p3 h2 recv: p1 p3 h1 recv: (nothing acks lost) h2 send: ack1 ack1 ack1 (dude, i missed a packet) h1 send: (nothing, waiting for ack) h2 send: (nothing, waiting for retransmit) h1 send: p1 p2 p3 (ack timed out) h2 send: (nothing, waiting for retransmit) what should happen is this: h1 send: p1 p2 p3 h2 recv: p1 p3 h1 recv: (nothing acks lost) h2 send: ack1 ack1 ack1 (dude, i missed a packet) h2 send: ack1 ack1 ack1 (dude, i missed a packet) h1 recv: ack1 ack1 ack1 h1 send: p2 p3 Basically, will the reciever keep acking not if 'it detects packet loss', but rather 'as long as packets are lost'. -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' http://www.morons.org/rants/gpl-harmful.php3 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 12:47:29PM -0800, Matthew Dillon wrote: Well, this is embarassing. I can reproduce this completely running 4.4-stable (Nov 17th kernel) on two machines. With newreno turned on, a TCP NFS mount only gets 80K/sec. With newreno turned off on the transmit side, a TCP NFS mount gets 7MB/sec. The state of the delayed-ack sysctl is irrelevant. This is without running any nfsiod's (which would mask the degredation of the synchronous messaging). I am tracking it down now. Is this the same problem that I experience on ssh connections between my 5.0-current laptop and my releng_4 server? When I run an 'ls' from the shell on large directories I get the response back block delay block delay block. I assumed that it was a problem with -current. Joe msg29413/pgp0.pgp Description: PGP signature
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
:what should happen is this: : :h1 send: p1 p2 p3 :h2 recv: p1 p3 : :h1 recv: (nothing acks lost) :h2 send: ack1 ack1 ack1 (dude, i missed a packet) : :h2 send: ack1 ack1 ack1 (dude, i missed a packet) :h1 recv: ack1 ack1 ack1 :h1 send: p2 p3 : :Basically, will the reciever keep acking not if 'it detects packet loss', :but rather 'as long as packets are lost'. : :-- :-Alfred Perlstein [[EMAIL PROTECTED]] Yuch. That won't help. Basically you are taking a brute-force approach Send the ack a whole bunch of times in case some of them get lost. Such an approach does not typically work very well. For example, if the packet loss occured due to link congestion your solution will actually make the link more congested rather then less. If there is significant latency in the path the acks can get into a following run with the transmitter, making the transmitter believe that the packet loss is worse then it actually is and responding in kind, resulting in even more incorrect acks. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
:Hi,
:
:I think that there are two different problems here. My situation
:involves a LAN (actually, a crossover cable).
:
:I have captured a trace of a 1 client run between the Linux driver and
:the FreeBSD test system as well as between the Linux driver and the same
:test system running Linux.
:
:I am noticing some interesting things. Linux uses the timestamp option
:in all the TCP segments I have looked at, so it is sending 12 more bytes
:per segment that FreeBSD.
:
:However, more interesting is that for small messages (less that 1460),
:FreeBSD does not seem to delay sending ACKs, so we get the following
:pattern:
:
:FREEBSD
:
:Driver - Test system: 94 byte IP DG with simulated command
:Test System - Driver: Ack after 83uS
:Test System - Driver: Psh Ack after 29uS with 79 total bytes in IP DG
:
:LINUX
:
:Driver - Test system: 106 byte IP DG with simulated command
:Test System - Driver: Psh Ack after 89uS with 91 total bytes in IP DG
:
:So, as you can see, Linux seems to shave some time off each transaction
:by avoiding sending extra ACKs.
:
:Also, what I am seeing is that neither FreeBSD nor Linux is doing ACK coalescing (if
:that is possible).
:
:
:While I understand that coalescing ACKs will mess up RTT calculations and SRTT a bit,
:it would serve to reduce the time taken until responses come back.
Hmm. If I ssh between two machines and use tcpdump to monitor
the packets I see proper delayed-ack operation:
(^A - a keystroke that is not echo'd, a delayed-ack occurs)
16:15:36.673259 216.240.41.12.1025 216.240.41.11.22: P 200:220(20) ack 181 win 17520
(DF) [tos 0x10]
16:15:36.70 216.240.41.11.22 216.240.41.12.1025: . ack 220 win 17520 (DF) [tos
0x10]
('a' - a keystroke that is echo'd, delayed-ack, ack is returned in echod data)
16:15:49.143239 216.240.41.12.1025 216.240.41.11.22: P 240:260(20) ack 181 win 17520
(DF) [tos 0x10]
16:15:49.156878 216.240.41.11.22 216.240.41.12.1025: P 181:201(20) ack 260 win 17520
(DF) [tos 0x10]
16:15:49.251975 216.240.41.12.1025 216.240.41.11.22: . ack 201 win 17520 (DF) [tos
0x10]
The timestamp could be creating an issue in your tests, though I'm not
entirely sure what the rules are for replies to timestamped packets.
Are you sure delayed acks are turned on on the FreeBSD box? An actual
tcpdump might be helpful here.
-Matt
To Unsubscribe: send mail to [EMAIL PROTECTED]
with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 11:49:13PM +, Josef Karthauser wrote: On Fri, Nov 30, 2001 at 03:45:21PM -0800, Matthew Dillon wrote: :... : I am tracking it down now. : :Is this the same problem that I experience on ssh connections between :my 5.0-current laptop and my releng_4 server? When I run an 'ls' :from the shell on large directories I get the response back block :delay block delay block. I assumed that it was a problem with :-current. : :Joe It sounds like the same problem. In fact, I seem to recall observing something very similar from my laptop while ssh'd into one of my servers, but at the time I though it was a hicup in the wireless network. Now though I think it was this same issue. I'm just about to reboot the server now with your recently committed changes - I'll let you know if it fixed anything. No, the problem remains after rebuilding the kernel on both boxes. Joe msg29421/pgp0.pgp Description: PGP signature
Re: FreeBSD performing worse than Linux?
: :No, the problem remains after rebuilding the kernel on both boxes. : :Joe Try to track down the sequence with a tcpdump. -Matt Matthew Dillon [EMAIL PROTECTED] To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: Found the problem, w/patch (was Re: FreeBSD performing worse than Linux?)
Quoting Matthew Dillon ([EMAIL PROTECTED]): The question here is... is it actually packet loss that is creating this issue for you and John, or is it something else? The only way to tell for sure is to run tcpdump on BOTH the client and server and then observe whether packet loss is occuring by comparing the dumps. Packet loss is the problem for sure. I am dumping on the server and client side. http://www.irbs.net/server-dump.html http://www.irbs.net/client-dump.html In 60Ms the server pushed out about 200 segments. My test writes 1 byte at a time on an existing ssh conection so the payload per segment is small, 48 bytes. (48 + IP + TCP) * 200 is around 17KB in 60Ms which probably overflowed the frame switch queue. The client is on a fractional T1, the server is on a 10Mb - OC3 connection 1200 network miles away. Jonathan Lemon pointed out in the TCP Anomalies thread that slow start seems to be broken. John Capo To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
* Matthew Dillon [EMAIL PROTECTED] [011130 17:45] wrote: :... : I am tracking it down now. : :Is this the same problem that I experience on ssh connections between :my 5.0-current laptop and my releng_4 server? When I run an 'ls' :from the shell on large directories I get the response back block :delay block delay block. I assumed that it was a problem with :-current. : :Joe It sounds like the same problem. In fact, I seem to recall observing something very similar from my laptop while ssh'd into one of my servers, but at the time I though it was a hicup in the wireless network. Now though I think it was this same issue. This may be a server problem, I'm ssh'd into a FreeBSD box from a NetBSD one and don't see the issue, going FreeBSD - FreeBSD at home seems to cause stalls, I'll try my netbsd laptop at home and see if i can reproduce the problem. I thought it was my crappy DSL causing the issue, perhaps not. -- -Alfred Perlstein [[EMAIL PROTECTED]] 'Instead of asking why a piece of software is using 1970s technology, start asking why software is ignoring 30 years of accumulated wisdom.' http://www.morons.org/rants/gpl-harmful.php3 To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Saturday, 1 December 2001 at 8:11:19 +1030, Richard Sharpe wrote: Matthew Dillon wrote: Well, this is embarassing. I can reproduce this completely running 4.4-stable (Nov 17th kernel) on two machines. With newreno turned on, a TCP NFS mount only gets 80K/sec. With newreno turned off on the transmit side, a TCP NFS mount gets 7MB/sec. The state of the delayed-ack sysctl is irrelevant. This is without running any nfsiod's (which would mask the degredation of the synchronous messaging). I have upgraded to 4.4-STABLE, and have hacked in some changes to ata-dma.c (provided by Greg Lehey, but I had to do it by hand) What did you have to do by hand? so my drive is now running at UDMA 100. Can you send me dmesg output? In particular, I had a printf output there to show what the BIOS had set. Background for other people: Richard has an IDE chip which claims to be a SiS 5591, which according to the data sheet can't do better than UDMA 33. When he runs Linux on the box, however, it claims to be running at UDMA 100, and this hack seems to have had the same effect. I have also ensured that disk write caching is on, which it seems to be by default in 4.4. Yes, I think this is correct. These changes have made a difference to the NetBench and dbench runs (improved them), but they have made no difference to the tbench runs, which only do network stuff. I'd like to see the new dbench results. The traffic in the tbench case is SMB taffic. Request/response, with a mixture of small requests and responses, and big request/small response or small request/big response, where big is 64K. I have switched off newreno, and it made no difference. I have switched off delayed_ack, and it reduced performance about 5 percent. I have made sure that SO_SNDBUF and SO_RCVBUF were set to 131072 (which seems to be the max), and it increased performance marginally (like about 2%), but consistently. Have you tried Matt Dillon's patch? I am still analysing the packet traces I have, but it seems to me that the crucial difference is Linux seems to delay longer before sending ACKs, and thus sends less ACKs. Since the ACK is piggybacked in the response (or the next request), it all works fine, and the reponse/request gets there sooner. However, I have not convinced myself that the saving of 20uS or so per request/response pair accounts for some 40+ Mb/s. As long as the ack traffic isn't saturating the link, and you're not running half-duplex, I can't see how that would be the problem. Greg -- See complete headers for address and phone numbers To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
I started noticing some TCP weirdness when I moved my bandwidth stats site from my office to my colo facility last week. The colo is five miles away by road and 1200 miles away by network. Netscape would stop for seconds at a time while loading the graph images but there was no consistency. Worked properly sometimes and sometimes not. Thanks for the much more detailed bug report vs. mine. Can you try disabling delayed acks to see if that helps, per another poster's response to this thread? sysctl net.inet.tcp.delayed_ack=0 Nate To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 10:09:58AM -0700, Nate Williams wrote: I started noticing some TCP weirdness when I moved my bandwidth stats site from my office to my colo facility last week. The colo is five miles away by road and 1200 miles away by network. Netscape would stop for seconds at a time while loading the graph images but there was no consistency. Worked properly sometimes and sometimes not. Thanks for the much more detailed bug report vs. mine. Can you try disabling delayed acks to see if that helps, per another poster's response to this thread? sysctl net.inet.tcp.delayed_ack=0 I tried this out and it made no difference whatsoever. I also tried out moving the same file via NFS and get transfer times that are within 5 seconds of the FTP times. I am beginning to suspect that I have a hardware issue here. Josh To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 12:42:34AM -0500, John Capo wrote: sent. find / -print | dd obs=1 will screw up within a few seconds and stay that way. Netstat in another ssh session shows data ready to go: Hmm, some ssh versions tend to hang randomly on lossy links (in the protocol perhaps, but I haven't ever tried to investigate this). Could you try the same in a telnet or rsh connection? I bet it will work. Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 01:03:54AM +0100, Pierre Beyssac wrote: On Thu, Nov 29, 2001 at 12:42:34AM -0500, John Capo wrote: sent. find / -print | dd obs=1 will screw up within a few seconds and stay that way. Netstat in another ssh session shows data ready to go: Hmm, some ssh versions tend to hang randomly on lossy links (in the protocol perhaps, but I haven't ever tried to investigate this). Could you try the same in a telnet or rsh connection? I bet it will work. Pierre This gives me the same 1.5megs/sec I am getting with ftp. Doesn't matter whether I use ssh or telnet. Josh To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
The behavior is different in a telnet session as is executing ssh server 'find / -print | dd obs=1'. Telnet pauses but not as long, the ssh command saturates my fractional T. What I am seeing may not be related to the issues Greg brought up at all. But it does seem odd to me that a window of data is queued on the server, the server has recevied the ack for the last byte it sent, but the server does not start sending until the client acks the same byte again with a data bearing segment. Don't know what the data bearing segment contains, ssh stuff maybe. Quoting Pierre Beyssac ([EMAIL PROTECTED]): On Thu, Nov 29, 2001 at 12:42:34AM -0500, John Capo wrote: sent. find / -print | dd obs=1 will screw up within a few seconds and stay that way. Netstat in another ssh session shows data ready to go: Hmm, some ssh versions tend to hang randomly on lossy links (in the protocol perhaps, but I haven't ever tried to investigate this). Could you try the same in a telnet or rsh connection? I bet it will work. Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 06:44:14PM +, Josh Paetzel wrote: Could you try the same in a telnet or rsh connection? I bet it will work. This gives me the same 1.5megs/sec I am getting with ftp. Doesn't matter whether I use ssh or telnet. Hmm, sorry, looks like I misunderstood John's initial mail then: I thought the ssh session was completely hung. Having said that, I looked more closely at the TCP session, and it looks like the transfer stops simply when the client's receive window is full. Looks quite normal to me. Client announces a window size of 17376: 21:41:46.418208 client.4427 server.22: . ack 11489 win 17376 nop,nop,timestamp 53827698 105528699 (DF) [tos 0x10] Server sends 1448 bytes: 21:41:47.460903 server.22 client.4427: . 11489:12937(1448) ack 144 win 17376 nop,nop,timestamp 105528895 53827698 (DF) [tos 0x10] Client announces a window size of 15928 (17376 - 1448): 21:41:47.569133 client.4427 server.22: . ack 12937 win 15928 nop,nop,timestamp 53827813 105528895 (DF) [tos 0x10] Client consumes some data internally then increases the window size by 1448 bytes (17376 - 15928): 21:41:49.001039 client.4427 server.22: P 144:192(48) ack 12937 win 17376 nop,nop,timestamp 53827954 105528895 (DF) [tos 0x10] Server sends just enough data to fill the window again: 21:41:49.001073 server.22 client.4427: . 28049:29497(1448) ack 192 win 17328 nop,nop,timestamp 105529049 53827954 (DF) [tos 0x10] This is consistent with the value in the backlog shown by netstat, What is the value of net.inet.tcp.sendspace on the server? It's 16K on -stable, it should be very interesting to try and increase it to 32K or 64K, it makes a lot difference on high bw*delay links. Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 02:25:47AM +0100, Pierre Beyssac wrote: This is consistent with the value in the backlog shown by netstat, What is the value of net.inet.tcp.sendspace on the server? It's 16K Uh, it wouldn't harm, but it won't do much good either on your example. Actually I should have said: increase net.inet.tcp.recvspace to 32K or 64K on the client instead :-) Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 02:25:47AM +0100, Pierre Beyssac wrote: What is the value of net.inet.tcp.sendspace on the server? It's 16K on -stable, it should be very interesting to try and increase it to 32K or 64K, it makes a lot difference on high bw*delay links. *grumble* I'll try to avoid my tirade. 16K _will cause problems for virtually all users_. You can do the math and see it won't keep a T1 full across country. Now that I understand the issues with increasing it I see why we can't go to real values (eg 256k) because FreeBSD's buffer management isn't up to the task. This default _really_ needs to be moved to 32k, with an increase in default MBUF's. Machines have enough RAM, and it will allow people to fill their home DSL connections. It also puts FreeBSD on par with other OS's. I also think we need to support some of the work on buffer management schemes to be able to do something even better. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Thu, Nov 29, 2001 at 08:42:25PM -0500, Leo Bicknell wrote: *grumble* I'll try to avoid my tirade. 16K _will cause problems for virtually all users_. You can do the math and see it won't keep a T1 full across country. I can't reproduce this result, 16K fills a T1 for 11 ms, which is 22000 km (at 2/3 of light speed), enough to get halfway round the earth... This default _really_ needs to be moved to 32k, with an increase in I'm not sure it is necessary. Remember that you can always use SO_RCVBUF/SO_SNDBUF to get bigger buffer sizes, bulk transfer tools can always set it explicitly to 64K if they like. For interactive tools like telnet or ssh, increasing the default size is not really useful. Interestingly enough, ftpd does set SO_SNDBUF, but from a quick glance at the code ftp doesn't seem to set SO_RCVBUF, and neither does fetch. Pierre To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
Re: FreeBSD performing worse than Linux?
On Fri, Nov 30, 2001 at 03:23:45AM +0100, Pierre Beyssac wrote: I can't reproduce this result, 16K fills a T1 for 11 ms, which is 22000 km (at 2/3 of light speed), enough to get halfway round the earth... Your math is a little funny. 4000 km New York to LA c = 300,000 km/sec Speed of light in fiber, approximately .66 c, or 198,000 km/sec. Approximate sum of buffering + serialization delay in the network, is a 15% penalty, or 168,300 kph. total speed. 4000 km one way == 8000 km two way, 8000 / 168300 = 47ms in my book, theoretial optimum. With an RTT of 47ms, you can move 16k per RTT, or or about 340k/sec. * If you find a cross country RTT of 47 ms I'll personally send you $20. around 60-65 is normal for good circuits, and 70-90 is not wholely unusual. * The 340k/sec assumes perfect network conditions, that is no dropped or delayed packets. Please search the archives. There are reams of information about this. -- Leo Bicknell - [EMAIL PROTECTED] - CCIE 3440 PGP keys at http://www.ufp.org/~bicknell/ Read TMBG List - [EMAIL PROTECTED], www.tmbg.org To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
RE: FreeBSD performing worse than Linux?
If you give me your IP address, I can ping *from* Columbia.edu to your machine and see what I get, that should pretty much solve any issues that may arise. -- Jonathan - Jonathan Slivko - Voyager Internet - www.voyageri.net Web Hosting - Web Design - Dedicated Services [EMAIL PROTECTED] - Phone: (877) 211-7842 -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]] On Behalf Of Ted Faber Sent: Thursday, November 29, 2001 10:27 PM To: Pierre Beyssac Cc: Leo Bicknell; Josh Paetzel; [EMAIL PROTECTED]; [EMAIL PROTECTED] Subject: Re: FreeBSD performing worse than Linux? On Fri, Nov 30, 2001 at 03:23:45AM +0100, Pierre Beyssac wrote: On Thu, Nov 29, 2001 at 08:42:25PM -0500, Leo Bicknell wrote: *grumble* I'll try to avoid my tirade. 16K _will cause problems for virtually all users_. You can do the math and see it won't keep a T1 full across country. I can't reproduce this result, 16K fills a T1 for 11 ms, which is 22000 km (at 2/3 of light speed), enough to get halfway round the earth... Not *this* Earth, what with real equipment and all. I get 22ms to my home machine less than 6 miles from my work machine via DSL. Both machines have very good connectivity, but there are delay artifacts in the world. And pinging ctr.columbia.edu from my machine in Southern California gets me 70 ms. Neither of those should show the DSL artifact. Columbia's not 9065 miles away, even by wire length. And, not to be too pedantic, you've calculated the 2/3c distance for 110ms, not 11 ms. :-) To Unsubscribe: send mail to [EMAIL PROTECTED] with unsubscribe freebsd-hackers in the body of the message
