On 29.05.2014 16:33, Nadav Har'El wrote:
Hi,
Hello!
I'm working on the OSv project (http://osv.io/), a new BSD-licensed
operating system for virtual machines. OSv's networking code is based
on that of FreeBSD.
I recently noticed an inefficiency that I believe exists also in
FreeBSD's networking code, and I was wondering why this was done,
and whether FreeBSD can also be improved in the same way by fixing
this problem.
My issue is that, for example, when running a UDP server answering
hundreds of thousands of requests per second, I get the same number of
calls to the routing table lookup function (rtalloc_ign_fib(), etc.).
These calls are relatively slow: Each involves several mutex locks and
unlocks (a rwlock for the radix tree, and a mutex for the individual
route), which are relatively slow in the uncontended case, but even worse
when several CPUs start to access the network heavily, and we start to see
context switches hurting the performance of the server even further.
Yes, that's true.
Looking at FreeBSD's udp_output(), I see it does the following:
error = ip_output(m, inp->inp_options, NULL, ipflags,
inp->inp_moptions, inp)
Note how NULL is passed as the third parameter. This tells ip_output
that it can't cache the previously found route, and needs to look for
it again and again on every packet output - even in the common case
where a socket will only ever send packets on one interface.
It seems that this change was done around FreeBSD 5.4. In the original
UCB code (4.4Lite), I see this:
error = ip_output(m, inp->inp_options, &inp->inp_route,
inp->inp_socket->so_options & (SO_DONTROUTE | SO_BROADCAST),
inp->inp_moptions);
So the last-found route was cached in inp->inp_route, and possibly
reused on the next packet to be sent.
Does anyone have any idea why inp->inp_route was removed in FreeBSD?
Doesn't this also hurt FreeBSD's network performance?
Well, there are two problems:
First, using cached routes makes it more complex to change routing stack
to be more efficient.
The second one is basically the fact that using cached routes is simply
incorrect:
no one protects/notifies you on interface removal. That's why we've
removed cached route support from various tunneling schemes.
There is some ongoing work to change rte_* api and eliminate the need
for per-rte mutex (and use different, more efficient lookup mechanisms).
There is also another alternative which you can currently use: flowtable
(not included in GENERIC).
It has been fixed recently and should work better in your case.
Thanks,
Nadav.
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "[email protected]"
