On Fri, Dec 03, 2010 at 01:12:57PM +0100, Claudio Jeker wrote: > On Fri, Dec 03, 2010 at 11:07:00PM +1300, Ben Aitchison wrote: > > On Wed, Dec 01, 2010 at 10:39:20PM -0500, Ted Unangst wrote: > > > yeah I found bumping to 64k made a big difference too, but for my > > > desktop, i have basically infinite memory, so there's little point > > > trying to find the right number. i went to 256k just to measure the > > > difference. but this isn't a long term fix. starting at 16k is ok, > > > as long as we can get the window bigger. > > > > > > That said, you realise you can still set the socket buffer size in an > > > > application - like squid, and relayd both support built in hard coding > > > > of > > > > socket buffer size. > > > > > > for a server, maybe i'd do so. but i'm not going around patching all > > > the client shit on my desktop (rebuild firefox!?) just for this. > > > > Hey I just discovered that relayd isn't raising the buffer sizes, whereas > > squid is. There may be some issue like not accepting the data quick enough > > causing issues? > > > > If I set a static buffer at 49152, I get the same speeds as my (raised) > > default. > > > > But if I use squid I get something more comparable to the speeds I get when > > I > > raise buffer to 262144. (which isn't 5 times as fast fwiw) > > > > So I'm assuming that some applications are not increasing the window size > > for > > some reason or other; > > > > Window size scaling is disabled when an application is issuing a > setsockopt() changing SO_SNDBUF or SO_RCVBUF.
Well I had been looking through the relayd code to figure out if it buffers, or how it buffers. And getting a bit confused, and figuring that libevent would do that kind of thing. Which I had only just started looking through. What I did find though was that relayd doesn't seem to set SO_SNDBUF/SO_RCVBUF unless it sets both with an explicit option. And it doesn't do so by default unless you add it to the configuration. The thing is not defining or setting it explicitally it behaved the same when Squid wasn't. Then, I suddenly realised that /sometimes/ it's working, and sometimes it's not. It was getting hard to test, with network congestion etc operating. And results weren't always predictable. What I did find though, was there was a MUCH higher incidence of speeds sticking around 344k/sec with the kernel changed to start it's window size at 49152 bytes starting point. But that sometimes it was getting a bit faster. But when operating from the box itself (which is ethernet connected) but relaying to a DSL connection the speed was higher. But also when going straight from the DSL to the box speed was also higher. And when relaying it was slower. Now my squid configuration has "read_ahead_buffer" set to 512k. Which is enough that it may be pulling away from being tightly bound to congestion fluctuations in either side. Giving higher average speeds. But I'm also wondering if congestion losses are making window size not fall below that size or something. And it's just because the link is slightly lossy. Something like 0.3% loss from memory. And maybe that's enough to drop window size increases. Thing is that's not local congestion. And transferring more doesn't necessarily make that worse. But that magic number kept happening. Oh and the latency of the remote site was 127msec so 1000/127 * 49152 / 1024 = 377.95k/sec. So it should have been going 377k/sec in ideal conditions. But I'm assuming there's some kind of extra latency above that. Anyway setting explicit socket buffer was faster. And I should probably revert to the default buffer sizes, and try and figure out why it's not increasing. But as it stands now setting an explicit buffer size 3x gives 3x speed increase through relayd, and using squid instead gives 3x to 4x speed boost (it's fluctuating mid way through connection, drops to 3x then raises again). Compared to not setting any socket options using relayd. Which seems broken to me. Also, even using Squid I'm getting better performance from a Linux box. Peaking at 2200k/sec. And dropping to 1826k/sec. Whereas the recv scaling on OpenBSD is only peaking at around 1450k/sec. I am not sure how Linux scales it's receive window, but it seems to be more aggressive. Ben.
