On Fri, Feb 14, 2020 at 6:25 PM Rudy <cra...@monkeybrains.net> wrote:
> On 2/12/20 7:21 PM, Rudy wrote: > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load (and > high interrupts). > > > Looking better! I made some changes based on BSDRP which I hadn't known > about -- I think ifqmaxlen was the tunable I overlooked. > > # > > https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local > net.link.ifqmaxlen="16384" > > This net.link.ifqmaxlen was set to help in case of lagg usage: I was not aware it could improve your use case. >From your first post, it looks like your setup is a 2 packages, 10 cores, 20 threads (disabled). And you have configured your Chelsio to use 16 queues (hw.cxgbe.Xrx=16): It's a good think to have a power of 2 number of queues with Chelsio, but I'm not sure it's a good idea to spread those queue across the 2 packages. So perhaps you should try: 1. To reduce queues to 8 queues and bind them to the local domain 2. Or keeping 16 queues, but re-enabling HyperThreading and bing them to the local domain too. (on -head with recent CPU and machdep.hyperthreading_intr_allowed, using hyper-threading improve forwarding performance). But anyway even with 16 queues spread over 2 domains, you should have better performance: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650v4_2x12Cores-Chelsio_T520-CR/hw.cxgbe.nXxq/results/fbsd12-stable.r354440.BSDRP.1.96/README.md Notice that I never monitoring the CPU load during my benches. Increasing the hw.cxgbe.holdoff_timer_idx was a good idea: I would expect lower interrupt usage too. Did you monitor the QPI link usage ? (kldload cpuctl && pcm-numa.x) _______________________________________________ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"