Thx Shawn for the detailed explanation. I wonder if such a fix should be put in mina-core too.
On Wed, Dec 16, 2009 at 05:20, <[email protected]> wrote: > Author: spearce > Date: Wed Dec 16 04:20:04 2009 > New Revision: 891122 > > URL: http://svn.apache.org/viewvc?rev=891122&view=rev > Log: > SSHD-54: Use OS default receive buffer to improve throughput > > MINA itself is actually forcing the receive buffer size to be 1024 > bytes, rather than the operating system default. By default the > NioAcceptor is created with a DefaultSocketSessionConfig [1] which > sets both the send and receive buffer sizes to 1024 bytes. > > NioSocketAcceptor [2] then turns around and sets this receive > buffer size into the accept socket, so all child sockets created > for incoming connections inherit this low default value. > > However, in the case of the send buffer, because the configured > value of 1024 matches the default of 1024 for the send buffer, > doSetAll [3] never actually changes the send buffer size away from > the operating system default. > > Unfortunately this means we have the following situation for our > NIO based sockets: > > MINA | OS Default > +------+----------- > RCV | 1024 | 43690 > SND | 8192 | 8192 > > So the receive buffer size under MINA is 42x smaller than the OS > default. But the send buffer size is the OS default, as MINA does > not directly set the send buffer size itself. So send throughput > is actually reasonable, but receive throughput is horrible. > > There is a good article [4] which explains how these settings impact > connection throughput. If we want any decent TCP/IP throughput we > need reasonably large buffers to increase the amount of data one > side can send before stopping and waiting for an ACK from the peer. > If our receive window is only 1024 bytes, our peer can't even send > a full TCP/IP packet on Ethernet without stopping and waiting for > an ACK from us. > > Of course one could try to argue that the MINA default of 1024 bytes > for the receive buffer is suitable, especially on a server with a > very large number of idle connections, or connections where a human > is typing all of the input data. But it is not suitable for a high > traffic data receiver, like the scp command. > > Unfortunately asking applications to raise the receive buffer size > on each individual connection through the SessionFactory doesn't > work on all platforms. On Linux with OpenJDK 6 the following > application code appears to work: > > setSessionFactory(new SessionFactory() { > �...@override > protected ServerSession createSession(final IoSession io) > throws Exception { > final SocketSessionConfig c = (SocketSessionConfig) io.getConfig(); > c.setKeepAlive(keepAlive); > c.setReceiveBufferSize(43690); > } > } > }); > > but the aggregate throughput is still horrible (3 MB/s). So the > receive buffer didn't actually increase with the peer. > > Linux seems to be capping the connection at the upper bound when > the connection starts. If we set the receive buffer size of the > NioSocketAcceptor to 43690, later drop the receive buffer size of > the individual connection to 1024 in the session factory, we can > later boost it when a specific command starts, and performance > behaves as expected. > > To match developer expectations we now reset the receive buffer > size to the operating system's default size before the server > starts accepting connections. This allows applications to get good > throughput by default, or raise/lower the receive buffer within > reasonable limits based on the command executed. > > References: > [1] > http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/DefaultSocketSessionConfig.java?view=markup#l44 > [2] > http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/nio/NioSocketAcceptor.java?view=markup#l248 > [3] > http://svn.apache.org/viewvc/mina/tags/2.0.0-RC1/core/src/main/java/org/apache/mina/transport/socket/AbstractSocketSessionConfig.java?view=markup#l58 > [4] http://www.ibm.com/developerworks/linux/library/l-hisock.html > > Modified: > mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java > > Modified: > mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java > URL: > http://svn.apache.org/viewvc/mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java?rev=891122&r1=891121&r2=891122&view=diff > ============================================================================== > --- mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java > (original) > +++ mina/sshd/trunk/sshd-core/src/main/java/org/apache/sshd/SshServer.java > Wed Dec 16 04:20:04 2009 > @@ -21,6 +21,7 @@ > import java.io.File; > import java.io.IOException; > import java.net.InetSocketAddress; > +import java.net.Socket; > import java.security.InvalidKeyException; > import java.security.PublicKey; > import java.util.ArrayList; > @@ -31,6 +32,9 @@ > import java.util.List; > import java.util.concurrent.CountDownLatch; > > +import org.slf4j.LoggerFactory; > +import org.slf4j.Logger; > + > import org.apache.mina.core.service.IoAcceptor; > import org.apache.mina.core.session.IoSession; > import org.apache.mina.core.session.IoSessionConfig; > @@ -107,6 +111,8 @@ > */ > public class SshServer extends AbstractFactoryManager implements > ServerFactoryManager { > > + private final Logger log = LoggerFactory.getLogger(getClass()); > + > protected IoAcceptor acceptor; > protected String host; > protected int port; > @@ -326,8 +332,26 @@ > > protected void configure(IoAcceptor acceptor) { > if (acceptor instanceof NioSocketAcceptor) { > - ((NioSocketAcceptor) acceptor).setReuseAddress(reuseAddress); > - ((NioSocketAcceptor) acceptor).setBacklog(backlog); > + final NioSocketAcceptor nio = (NioSocketAcceptor) acceptor; > + nio.setReuseAddress(reuseAddress); > + nio.setBacklog(backlog); > + > + // MINA itself forces our socket receive buffer to 1024 bytes > + // by default, despite what the operating system defaults to. > + // This limits us to about 3 MB/s incoming data transfer. By > + // forcing back to the operating system default we can get a > + // decent transfer rate again. > + // > + final Socket s = new Socket(); > + try { > + try { > + > nio.getSessionConfig().setReceiveBufferSize(s.getReceiveBufferSize()); > + } finally { > + s.close(); > + } > + } catch (IOException e) { > + log.warn("cannot adjust SO_RCVBUF back to system default", > e); > + } > } > if (sessionConfig != null) { > acceptor.getSessionConfig().setAll(sessionConfig); > > > -- Cheers, Guillaume Nodet ------------------------ Blog: http://gnodet.blogspot.com/ ------------------------ Open Source SOA http://fusesource.com
