> Date: Thu, 23 Jun 2016 13:09:28 +0200
> From: Alexander Bluhm <alexander.bl...@gmx.net>
>
> On Wed, Jun 22, 2016 at 10:54:27PM +1000, David Gwynne wrote:
> > secondly, allocating more than 4g at a time to socket buffers is
> > generally a waste of memory. in practice you should scale the amount
> > of memory available to sockets according to the size of the tcp
> > windows you need to saturate the bandwidth available to the box.
>
> Currently OpenBSD limits the socket buffer size to 256k.
> #define SB_MAX (256*1024) /* default for max chars in sockbuf */
>
> For downloading large files from the internet this is not sufficinet
> anymore. After customer complaints we have increased the limit to
> 1MB. This still does not give maximum throughput, but granting
> more could easily result in running out of mbufs. 16MB would be
> sufficent.
>
> Besides from single connections with high throughput we also have
> a lot of long running connections, say some 10000. Each connection
> over a relay needs two sockets and four socket buffers. With 1MB
> limit and 10000 connections the theoretical maximum is 40GB.
>
> It is hard to figure out which connections need socket buffer space
> in advance. tcp_update_{snd,rcv}space() adjusts it dynamically,
> there sbchecklowmem() has a first come first serve policy. Another
> challenge is, that the peers on both sides of the relay can decide
> wether they fill our buffers.
>
> Besides from finding a smarter algorithm to distribute the socket
> buffer space, increasing the number of mbufs could be a solution.
> Our server machines mostly relay connection data, there I seems
> seductive to use much more mbuf memory to speed up TCP connetions.
> Without 64 bit DMA most memory of the machine is unused.
>
> Also modern BIOS maps only 2GB in low region. All DMA devices must
> share these. Putting mbufs high should reduce pressure.
>
> Of course there are problems with network adaptors that support
> less DMA space and with hotplug configurations. For a general
> solution we can implement bounce buffers, disable the feature on
> such machines or have a knob.
We really don't want to implement bounce-buffers. Adding IOMMU
support is probably a better approach as it also brings some security
benefits. Not all amd64 hardware supports an IOMMU. And hardware
that does support it doesn't always have it enabled. But for modern
hardware an iommu is pretty much standard, except for the absolute
low-end. But those low-end machines tend to have only 2GB of memory
anyway.
