On Fri, Feb 14, 2020 at 6:25 PM Rudy <cra...@monkeybrains.net> wrote:

> On 2/12/20 7:21 PM, Rudy wrote:
>  > I'm having issues with a box that is acting as a BGP router for my
> network.  3 Chelsio cards, two T5 and one T6.  It was working great
> until I turned up our first port on the T6.  It seems like traffic
> passing in from a T5 card and out the T6 causes a really high load (and
> high interrupts).
> Looking better!  I made some changes based on BSDRP which I hadn't known
> about -- I think ifqmaxlen was the tunable I overlooked.
> #
> https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local
> net.link.ifqmaxlen="16384"
This net.link.ifqmaxlen was set to help in case of lagg usage: I was not
aware it could improve your use case.

>From your first post, it looks like your setup is a 2 packages, 10 cores,
20 threads (disabled).
And you have configured your Chelsio to use 16 queues (hw.cxgbe.Xrx=16):
It's a good think to have a power of 2 number of queues with Chelsio, but
I'm not sure it's a good idea to spread those queue across the 2 packages.
So perhaps you should try:
1. To reduce queues to 8 queues and bind them to the local domain
2. Or keeping 16 queues, but re-enabling HyperThreading and bing them to
the local domain too. (on -head with recent CPU
and machdep.hyperthreading_intr_allowed, using hyper-threading improve
forwarding performance).

But anyway even with 16 queues spread over 2 domains, you should have
better performance:

Notice that I never monitoring the CPU load during my benches.
Increasing the hw.cxgbe.holdoff_timer_idx was a good idea: I would expect
lower interrupt usage too.

Did you monitor the QPI link usage ? (kldload cpuctl && pcm-numa.x)
freebsd-net@freebsd.org mailing list
To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"

Reply via email to