On Fri, Sep 02, 2016 at 13:43 +0200, Alexander Bluhm wrote: > On Fri, Sep 02, 2016 at 10:33:19AM +0200, Andreas Bartelt wrote: > > On 09/02/16 10:24, Alexander Bluhm wrote: > > > I see a performance drop to 10 Mbit/sec on some old i386 machines > > > with em(4). Can you try this kernel diff to see wether it is the > > > same problem? > > > > > > > yes, >50 MB/s now. Thanks! > > So I think this happens: > > sosend() uses m_getuio() now to allocate a MAXMCLBYTES mbuf cluster, > that is 65536 bytes. sbreserve() calculates the upper cluster limit > min(cc * 2, sb_max + (sb_max / MCLBYTES) * MSIZE). In our case it > is min(1024*16 * 2, (256*1024) + ((256*1024) / (1 << 11)) * 256) > == min(32768, 294912) == 32768. > > So after allocating a single mbuf cluster the sending socket buffer > has no space anymore. As tcp_output() keeps the mbuf cluster for > retransmits, it will be freed only after all ACKs have been received. > That kills performance totally. > > To allow cycling through the mbufs periodically, I think we need > space for at least 3 of them. Note that this diff also affects the > mbuf size on the receiving side, but I think it does not matter > much as the data size is also limited. > > Andreas, can you revert the diff I sent previously and try this one > instead? > > ok? > > bluhm >
I see the same problem on large MTU (9000) networks now. s/MAXMCLBYTES/PAGE_SIZE/ in m_getuio fixes it again for me, while bumping 3 to 6 in sbreserve doesn't. Can somebody confirm this to rule out some local issues? > Index: kern/uipc_socket2.c > =================================================================== > RCS file: /data/mirror/openbsd/cvs/src/sys/kern/uipc_socket2.c,v > retrieving revision 1.64 > diff -u -p -r1.64 uipc_socket2.c > --- kern/uipc_socket2.c 28 Jun 2016 14:47:00 -0000 1.64 > +++ kern/uipc_socket2.c 2 Sep 2016 10:29:10 -0000 > @@ -397,7 +397,8 @@ sbreserve(struct sockbuf *sb, u_long cc) > if (cc == 0 || cc > sb_max) > return (1); > sb->sb_hiwat = cc; > - sb->sb_mbmax = min(cc * 2, sb_max + (sb_max / MCLBYTES) * MSIZE); > + sb->sb_mbmax = max(3 * MAXMCLBYTES, > + min(cc * 2, sb_max + (sb_max / MCLBYTES) * MSIZE)); > if (sb->sb_lowat > sb->sb_hiwat) > sb->sb_lowat = sb->sb_hiwat; > return (0);
