On Mon, Oct 24, 2016 at 19:04 +0200, Hrvoje Popovski wrote:
> Hi all,
>
> OpenBSD box acts as transit router for /8 networks without pf and with
> this sysctls
>
> ddb.console=1
> kern.pool_debug=0
> net.inet.ip.forwarding=1
> net.inet.ip.ifq.maxlen=8192
>
> netstat
> 11/8 192.168.11.2 UGS 0 114466419 - 8 ix0
> 12/8 192.168.12.2 UGS 0 0 - 8 ix1
> 13/8 192.168.13.2 UGS 0 0 - 8 myx0
> 14/8 192.168.14.2 UGS 0 0 - 8 myx1
> 15/8 192.168.15.2 UGS 0 0 - 8 em3
> 16/8 192.168.16.2 UGS 0 89907239 - 8 em2
> 17/8 192.168.17.2 UGS 0 65791508 - 8 bge0
> 18/8 192.168.18.2 UGS 0 0 - 8 bge1
>
> while testing dlg@ "mcl2k2 mbuf clusters" patch with todays -current i
> saw that performance with plain -current drops for about 300Kpps vs
> -current from 06.10.2016. by bisecting cvs tree it seems that this
> commit is guilty for this
>
> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/net/if_ethersubr.c?rev=1.240&content-type=text/x-cvsweb-markup
>
I don't see how this change can affect performance in such a way
unless you're sending jumbo packets, but then the packet rates
are too high. Are you 100% sure it's this particular change?
What kind of traffic are you testing this with?
I assume small IP or UDP packets, correct?
>
> -current from 24.10.2016
>
> ix
> send receive
> 690Kpps 690Kpps
> 700Kpps 680Kpps
> 800Kpps 350Kpps
> 1.4Mpps 305Kpps
> 14Mpps 305Kpps
>
> em
> send receive
> 690Kpps 690Kpps
> 700Kpps 680Kpps
> 800Kpps 700Kpps
> 1.4Mpps 700Kpps
>
> bge
> send receive
> 620Kpps 620Kpps
> 630Kpps 515Kpps
> 700Kpps 475Kpps
> 800Kpps 430Kpps
> 1.4Mpps 350Kpps
>
>
> -current with if_ethersubr.c version 1.239
>
> ix
> send receive
> 910Kpps 910Kpps
> 920Kpps 820Kpps
> 1Mpps 825Kpps
> 1.4Mpps 700Kpps
> 14Mpps 700Kpps
>
> em
> send receive
> 940Kpps 940Kpps
> 950Kpps 845Kpps
> 1Mpps 855Kpps
> 1.4Mpps 845Kpps
>
> bge
> send receive
> 620Kpps 620Kpps
> 630Kpps 525Kpps
> 700Kpps 485Kpps
> 800Kpps 435Kpps
> 1.4Mpps 350Kpps
>
>
> applying dlg@ "mcl2k2 mbuf clusters" patch to todays -current brings
> performance back to ix and em ... bge is emotional as always :)
>
> ix
> send receive
> 900Kpps 900Kpps
> 910Kpps 895Kpps
> 1Mpps 895Kpps
> 1.4Mpps 810Kpps
> 14Mpps 815Kpps
>
> em
> send receive
> 940Kpps 940Kpps
> 950Kpps 930Kpps
> 1Mpps 920Kpps
> 1.4Mpps 930Kpps
>
> bge
> send receive
> 620Kpps 620Kpps
> 630Kpps 520Kpps
> 700Kpps 475Kpps
> 800Kpps 430Kpps
> 1.4Mpps 366Kpps
>
>
>
> i honestly don't know what all that means, i'm just writing my
> observation ...
>
Actually I'd like to know what causes this.
So far I've noticed that the code generating ICMP error doesn't
reserve space for the link header but it's unlikely a culprit.
(The diff was only compile tested so far...)
diff --git sys/netinet/ip_icmp.c sys/netinet/ip_icmp.c
index cdd60aa..b3ddea4 100644
--- sys/netinet/ip_icmp.c
+++ sys/netinet/ip_icmp.c
@@ -208,19 +208,21 @@ icmp_do_error(struct mbuf *n, int type, int code,
u_int32_t dest, int destmtu)
if (icmplen + ICMP_MINLEN > MCLBYTES)
icmplen = MCLBYTES - ICMP_MINLEN - sizeof (struct ip);
m = m_gethdr(M_DONTWAIT, MT_HEADER);
- if (m && (sizeof (struct ip) + icmplen + ICMP_MINLEN > MHLEN)) {
+ if (m && (max_linkhdr + sizeof(struct ip) + icmplen +
+ ICMP_MINLEN > MHLEN)) {
MCLGET(m, M_DONTWAIT);
if ((m->m_flags & M_EXT) == 0) {
m_freem(m);
m = NULL;
}
}
if (m == NULL)
goto freeit;
+ m->m_data += max_linkhdr;
/* keep in same rtable */
m->m_pkthdr.ph_rtableid = n->m_pkthdr.ph_rtableid;
m->m_len = icmplen + ICMP_MINLEN;
if ((m->m_flags & M_EXT) == 0)
MH_ALIGN(m, m->m_len);
