On Wed, 26 Apr 2006, Stephen Clark wrote:
Sorry not to have caught this thread earlier; I've been on travel for the
last few weeks. My general suggestion would be to try to narrow the code
paths traversed to try to eliminate as much code as possible from the
search. It sounds like you've done that pretty effectively :-).
Typically, memory leaks occur in edge error cases, where the memory is not
properly released, or ownership is unclear. My suggestion would be to add
counters (or look at existing counters where already present) and see if
there's an error case being triggered in about the same quantity that mbuf
leakage is occuring. Chances are, there's an error being returned and a
missing m_freem().
Based on your comments above, I might also pay attention to the routing
socket path -- the rate of leak could correspond to the routing daemons
talking to the network stack, rather than the rate of traffic. For
example, it could be that one of the routing messages is handled improperly
resulting in a leak.
Unfortunately, tracking down memory leaks can be quite difficult, and tends
to require a combination of dogged persistence and luck...
Good news and bad news.
I managed to get enough of our system running on 6.x stable to test and it
does not appear to lose mbufs. Bad news my ipsec transfer rate dropped from
54mbits/sec to 39mbits/sec. We need to be able to handle a t3 (45mbits/sec).
Any ideas as to why this drop off in 6.x?
Well, that's good about the good news, but not so great about the bad news.
Are you using IPv6? If not, could you look at how FAST_IPSEC performs instead
of the default IPSEC package, if you're not already doing so? The KAME IPSEC
implementation is not multi-processor compatible, so when IPSEC support is
compiled into the kernel, execution of the network stack and related
components is limited to a single CPU (you'll see a warning about this at boot
if this is the case). This is something we hope to fix in a future release,
but in the mean time FAST_IPSEC offers improved performance and SMP
scalability, as well as support for hardware crypto acceleration. The
downside to FAST_IPSEC is that it doesn't currently support IPv6, which is
also something we'd like to fix in the future.
Aside from the above, there are a number of other things we can look at to
decide what the source of the performance problem is. First, I'd encourage
you to make sure any system debugging features, such as INVARIANTS,
INVARIANT_SUPPORT, and WITNESS, are disabled. Next, I would recommend
generating a high level of your load on the system, and using systat -vmstat 1
and top -S to grab some information about it in the steady state. I recommend
letting both run for a couple of minutes, then grabbing the output from both
of them. This will give us an idea of where CPU usage is going in the kernel,
which is where I assume most of your workload ends up being handled.
Thanks,
Robert N M Watson
_______________________________________________
[email protected] mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "[EMAIL PROTECTED]"
