> Hi, > > On Thu, Sep 09, 2010 at 06:32:14PM -0500, open...@rkmorris.us wrote: > >> I can't explain the information below, but hopefully someone a whole >> bunch smarter than me understands it - as it seems to help with >> TCP throughput in OpenVPN (the holy grail??? ... :-)). >> > > Did you change TCP window size on the client, or on the OpenVPN process? > > If changing the TCP window size on the client helps, this is somewhat > unsurprising - OpenVPN adds delay, and TCP with a too-small window size > breaks down miserably if the delay goes up. > > What's the "default" TCP window size here? > > Given that you can even see an effect in the "direct" case, I suspect that > your machines' TCP window sizes are *way* too small - and this has not much > to do with TCP-over-TCP, but you would see the same effect for UDP-based > OpenVPN as well. > > Simple example, to clarify the effect of round trip time (delay) on > TCP throughput: > > TCP uses a sliding window, that is, the sender can send as many packets > as the "window size" permits before the sender has to wait for an acknowledge > from the receiver. Now, if the link has a noticeable delay AND lots of > bandwidth, the bandwidth * time product adds up to the minimum window size > you need to use: > > 1000 Mbit/s link > 1 ms delay (1/1000s) > > 1000 Mbit/s * 1ms = 1 Mbit = 100 Kbyte "in flight" - that is, if the sender > keeps sending packets back to back, with no pause in between, it needs to > send out 100 kbyte before the first ACK packet comes back. > > If the sender only sends 10 kbyte because the TCP window size doesn't permit > sending more, these 10 kbyte are sent in ~ 0.1 ms, and then it has to > wait for 0.9 ms for the ACKs to come in - send again for 0.1 ms, wait 0.9 ms > (roughly). So it's only using 10% of the available sending capacity, and > the performance sucks. > > > So, in your examples, my guess is that the TCP window size is "barely large > enough" for the "direct" case (but the link is not fully saturated unless > you increase the window size) - and since OpenVPN adds some noticeable > delay, the bandwidth * delay product goes up, and so needs the window size > to be able to fully saturate the link... > I wonder if the iPerf results only apply to iPerf tests... Linux autotunes TCP parameters, including send and receive windows[1]. However, increasing the maximum TCP buffer / memory size may be useful sometimes [2]. Also, an application can choose to use a smaller buffer than the OS default [2]: apparently iPerf does this. Windows also autotunes it's TCP parameters[3], although manual editing is an option[4].
Russell: perhaps you could try tuning the OS-level TCP buffer size and then test the results with a few different real-world applications (e.g. netcat, scp)? [1] <http://www.psc.edu/networking/projects/tcptune/#Linux> [2] <http://www.psc.edu/networking/projects/tcptune/#options> [3] <http://fasterdata.es.net/TCP-tuning/> [4] <http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html> -- Samuli Seppänen Community Manager OpenVPN Technologies, Inc irc freenode net: mattock
