Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 2/14/20 2:58 PM, BulkMailForRudy wrote: pmcstat -S inst_retired.any -Tw1 I had time to run pmcstat during a larger TX load on Chelsio T6 card. I'm loosing a hope on using FreeBSD as one of my edge routers outside of 10Gbps ports. *Start (been running for 4 days), cc1: RX 2.3Gbps / 130Mbps* %SAMP IMAGE FUNCTION CALLERS 20.4 kernel sched_idletd fork_exit 14.1 kernel cpu_search_highest cpu_search_highest:11.9 sched_switch:1.3 sched_idletd:0.9 5.5 kernel cpu_search_lowest cpu_search_lowest:5.0 sched_pickcpu:0.5 4.6 kernel eth_tx drain_ring 3.7 kernel rn_match fib4_lookup_nh_basic 2.0 kernel epoch_enter_preempt arpresolve 2.0 kernel mac_ifnet_check_tran ether_output 1.9 kernel lock_delay __mtx_lock_sleep 1.7 kernel service_iq_fl t4_intr 1.6 kernel ip_tryforward ip_input 1.5 kernel get_scatter_segment service_iq_fl 1.5 kernel mac_ifnet_create_mbu ether_nh_input 1.4 kernel parse_pkt cxgbe_transmit 1.3 kernel netisr_dispatch_src ether_demux:0.7 ether_input:0.6 1.3 kernel cxgbe_transmit ether_output_frame 1.2 kernel _rm_rlock in_localip 1.2 kernel mp_ring_enqueue cxgbe_transmit 1.2 kernel ether_output ip_tryforward *5 minutes in sending more traffic out cc1, cc1: RX 2.3Gbps / TX 1.7Gbps* %SAMP IMAGE FUNCTION CALLERS 17.0 kernel sched_idletd fork_exit 15.3 kernel cpu_search_highest cpu_search_highest:13.0 sched_switch:1.3 sched_idletd:1.0 6.9 kernel cpu_search_lowest cpu_search_lowest:6.3 sched_pickcpu:0.6 4.8 kernel eth_tx drain_ring 3.7 kernel rn_match fib4_lookup_nh_basic 2.7 kernel lock_delay __mtx_lock_sleep 2.0 kernel mac_ifnet_check_tran ether_output 1.9 kernel epoch_enter_preempt arpresolve 1.8 kernel service_iq_fl t4_intr 1.6 kernel mac_ifnet_create_mbu ether_nh_input 1.6 kernel ip_tryforward ip_input *10 minutes in, attempting to make cc1 referred outbound for whole network (~ 5Gbps), and the box starts dropping packets:* PMC: [inst_retired.any] Samples: 1019841 (100.0%) , 19068 unresolved %SAMP IMAGE FUNCTION CALLERS 24.8 kernel cpu_search_highest cpu_search_highest:22.7 sched_idletd:1.6 6.2 kernel lock_delay __mtx_lock_sleep 6.0 kernel rn_match fib4_lookup_nh_basic 5.7 kernel eth_tx drain_ring 2.7 kernel sched_idletd fork_exit 2.6 kernel tdq_move sched_idletd 2.5 kernel mac_ifnet_check_tran ether_output 2.1 kernel mac_ifnet_create_mbu ether_nh_input 2.0 kernel epoch_enter_preempt arpresolve *Reverting to ~ 150Mbps TX on cc1* PMC: [inst_retired.any] Samples: 136053 (100.0%) , 2767 unresolved %SAMP IMAGE FUNCTION CALLERS 27.1 kernel sched_idletd fork_exit 14.1 kernel cpu_search_highest cpu_search_highest:11.5 sched_switch:1.6 sched_idletd:0.9 4.4 kernel cpu_search_lowest cpu_search_lowest 4.1 kernel eth_tx drain_ring 3.4 kernel rn_match fib4_lookup_nh_basic 1.8 kernel epoch_enter_preempt arpresolve 1.8 kernel mac_ifnet_check_tran ether_output 1.6 kernel service_iq_fl t4_intr 1.4 kernel ip_tryforward ip_input 1.4 kernel get_scatter_segment service_iq_fl 1.3 kernel mac_ifnet_create_mbu ether_nh_input 1.3 kernel sched_switch mi_switch 1.3 kernel parse_pkt cxgbe_transmit 1.2 kernel _rm_rlock in_localip ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 2/14/20 4:21 AM, Andrey V. Elsukov wrote:
On 13.02.2020 06:21, Rudy wrote:
I'm having issues with a box that is acting as a BGP router for my
network. 3 Chelsio cards, two T5 and one T6. It was working great
until I turned up our first port on the T6. It seems like traffic
passing in from a T5 card and out the T6 causes a really high load (and
high interrupts).
Traffic (not that much, right?)
Dev RX bps TX bps RX PPS TX PPS Error
cc0 0 0 0 0 0
cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink,
filtering inbound routes to keep TX low)
cxl0 287 k 2015 M 353 244 k 0 (our network)
cxl1 940 M 3115 M 176 k 360 k 0 (our network)
cxl2 634 M 1014 M 103 k 128 k 0 (our network)
cxl3 1 k 16 M 1 4 k 0
cxl4 0 0 0 0 0
cxl5 0 0 0 0 0
cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0)
cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0)
ixl0 913 k 18 M 0 0 0
ixl1 1 M 30 M 0 0 0
lagg0 4019 M 1554 M 491 k 271 k 0
lagg1 1 M 48 M 0 0 0
FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg
1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827
Hi,
did you try to use pmcstat to determine what is the heaviest task for
your system?
# kldload hwpmc
# pmcstat -S inst_retired.any -Tw1
PMC: [inst_retired.any] Samples: 168557 (100.0%) , 2575 unresolved
Key: q => exiting...
%SAMP IMAGE FUNCTION CALLERS
16.6 kernel sched_idletd fork_exit
14.7 kernel cpu_search_highest cpu_search_highest:12.4
sched_switch:1.4 sched_idletd:0.9
10.5 kernel cpu_search_lowest cpu_search_lowest:9.6
sched_pickcpu:0.9
4.2 kernel eth_tx drain_ring
3.4 kernel rn_match fib4_lookup_nh_basic
2.4 kernel lock_delay __mtx_lock_sleep
1.9 kernel mac_ifnet_check_tran ether_output
Then capture several first lines from the output and quit using 'q'.
Do you use some firewall? Also, can you show the snapshot from the `top
-HPSIzts1` output.
last pid: 28863; load averages: 9.30, 10.33,
10.56 up 0+14:16:08 14:53:23
817 threads: 25 running, 586 sleeping, 206 waiting
CPU 0: 0.8% user, 0.0% nice, 6.2% system, 0.0% interrupt, 93.0% idle
CPU 1: 2.4% user, 0.0% nice, 0.0% system, 7.9% interrupt, 89.8% idle
CPU 2: 0.0% user, 0.0% nice, 0.8% system, 7.1% interrupt, 92.1% idle
CPU 3: 1.6% user, 0.0% nice, 0.0% system, 10.2% interrupt, 88.2% idle
CPU 4: 0.0% user, 0.0% nice, 0.0% system, 9.4% interrupt, 90.6% idle
CPU 5: 0.8% user, 0.0% nice, 0.8% system, 20.5% interrupt, 78.0% idle
CPU 6: 1.6% user, 0.0% nice, 0.0% system, 5.5% interrupt, 92.9% idle
CPU 7: 0.0% user, 0.0% nice, 0.0% system, 3.1% interrupt, 96.9% idle
CPU 8: 0.8% user, 0.0% nice, 0.8% system, 7.1% interrupt, 91.3% idle
CPU 9: 0.0% user, 0.0% nice, 0.8% system, 9.4% interrupt, 89.8% idle
CPU 10: 0.0% user, 0.0% nice, 0.0% system, 35.4% interrupt, 64.6% idle
CPU 11: 0.0% user, 0.0% nice, 0.0% system, 36.2% interrupt, 63.8% idle
CPU 12: 0.0% user, 0.0% nice, 0.0% system, 38.6% interrupt, 61.4% idle
CPU 13: 0.0% user, 0.0% nice, 0.0% system, 49.6% interrupt, 50.4% idle
CPU 14: 0.0% user, 0.0% nice, 0.0% system, 46.5% interrupt, 53.5% idle
CPU 15: 0.0% user, 0.0% nice, 0.0% system, 32.3% interrupt, 67.7% idle
CPU 16: 0.0% user, 0.0% nice, 0.0% system, 46.5% interrupt, 53.5% idle
CPU 17: 0.0% user, 0.0% nice, 0.0% system, 56.7% interrupt, 43.3% idle
CPU 18: 0.0% user, 0.0% nice, 0.0% system, 31.5% interrupt, 68.5% idle
CPU 19: 0.0% user, 0.0% nice, 0.8% system, 34.6% interrupt, 64.6% idle
Mem: 636M Active, 1159M Inact, 5578M Wired, 24G Free
ARC: 1430M Total, 327M MFU, 589M MRU, 32K Anon, 13M Header, 502M Other
268M Compressed, 672M Uncompressed, 2.51:1 Ratio
Swap: 4096M Total, 4096M Free
PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND
12 root -92 - 0B 3376K WAIT 13 41:13 12.86%
intr{irq358: t5nex0:2a1}
12 root -92 - 0B 3376K WAIT 12 48:08 12.77%
intr{irq347: t5nex0:1a6}
12 root -92 - 0B 3376K CPU13 13 47:40 11.96%
intr{irq348: t5nex0:1a7}
12 root -92 - 0B 3376K WAIT 17 43:46 11.38%
intr{irq342: t5nex0:1a1}
12 root -92 - 0B 3376K WAIT 14 29:17 10.70%
intr{irq369: t5nex0:2ac}
12 root -92 - 0B 3376K WAIT 11 47:55 9.85%
intr{irq428: t5nex1:2a5}
12 root -92 - 0B 3376K WAIT 16 46:11 9.22%
intr{irq351: t5nex0:1aa}
12 root -92 - 0B
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 2/14/20 10:00 AM, Olivier Cochard-Labbé wrote: On Fri, Feb 14, 2020 at 6:25 PM Rudy wrote: On 2/12/20 7:21 PM, Rudy wrote: > I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Looking better! I made some changes based on BSDRP which I hadn't known about -- I think ifqmaxlen was the tunable I overlooked. # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" This net.link.ifqmaxlen was set to help in case of lagg usage: I was not aware it could improve your use case. oThanks for the feedback. Maybe it was a coincidence. Load has creep back up to 15. From your first post, it looks like your setup is a 2 packages, 10 cores, 20 threads (disabled). And you have configured your Chelsio to use 16 queues (hw.cxgbe.Xrx=16): It's a good think to have a power of 2 number of queues with Chelsio, but I'm not sure it's a good idea to spread those queue across the 2 packages. So perhaps you should try: 1. To reduce queues to 8 queues and bind them to the local domain 2. Or keeping 16 queues, but re-enabling HyperThreading and bing them to the local domain too. (on -head with recent CPU and machdep.hyperthreading_intr_allowed, using hyper-threading improve forwarding performance). But anyway even with 16 queues spread over 2 domains, you should have better performance: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650v4_2x12Cores-Chelsio_T520-CR/hw.cxgbe.nXxq/results/fbsd12-stable.r354440.BSDRP.1.96/README.md OK, I can work on the chelsio_affinity script. hour later ... OK, tested and updated on github. Notice that I never monitoring the CPU load during my benches. Increasing the hw.cxgbe.holdoff_timer_idx was a good idea: I would expect lower interrupt usage too. I've have some standard SNMP monitoring and can correlate the load spinning out of control to ping loss and packet loss. # vmstat -i | tail -1 Total 12217353774 324329 Did you monitor the QPI link usage ? (kldload cpuctl && pcm-numa.x) I haven't. I'll look into that. Hoping the numa-domain locking helps. Currently I have things bound to the right domain, just need to shrink the queue size and reboot! irq289: t6nex0:err:261 @cpu0(domain0): 0 irq290: t6nex0:evt:263 @cpu0(domain0): 4 irq291: t6nex0:0a0:265 @cpu1(domain0): 0 irq292: t6nex0:0a1:267 @cpu2(domain0): 0 irq293: t6nex0:0a2:269 @cpu3(domain0): 0 irq294: t6nex0:0a3:271 @cpu4(domain0): 0 irq295: t6nex0:0a4:273 @cpu5(domain0): 0 irq296: t6nex0:0a5:275 @cpu6(domain0): 0 irq297: t6nex0:0a6:277 @cpu7(domain0): 0 irq298: t6nex0:0a7:279 @cpu8(domain0): 0 irq299: t6nex0:0a8:281 @cpu9(domain0): 0 irq300: t6nex0:0a9:283 @cpu1(domain0): 0 irq301: t6nex0:0aa:285 @cpu2(domain0): 0 irq302: t6nex0:0ab:287 @cpu3(domain0): 0 irq303: t6nex0:0ac:289 @cpu4(domain0): 0 irq304: t6nex0:0ad:291 @cpu5(domain0): 0 irq305: t6nex0:0ae:293 @cpu6(domain0): 0 irq306: t6nex0:0af:295 @cpu7(domain0): 0 irq307: t6nex0:1a0:297 @cpu8(domain0): 185404641 irq308: t6nex0:1a1:299 @cpu9(domain0): 146802111 irq309: t6nex0:1a2:301 @cpu1(domain0): 133930820 irq310: t6nex0:1a3:303 @cpu2(domain0): 173156318 irq311: t6nex0:1a4:305 @cpu3(domain0): 132151349 irq312: t6nex0:1a5:307 @cpu4(domain0): 149108252 irq313: t6nex0:1a6:309 @cpu5(domain0): 149196634 irq314: t6nex0:1a7:311 @cpu6(domain0): 184211395 irq315: t6nex0:1a8:313 @cpu7(domain0): 151266056 irq316: t6nex0:1a9:315 @cpu8(domain0): 169259534 irq317: t6nex0:1aa:317 @cpu9(domain0): 164117244 irq318: t6nex0:1ab:319 @cpu1(domain0): 157471862 irq319: t6nex0:1ac:321 @cpu2(domain0): 127662140 irq320: t6nex0:1ad:323 @cpu3(domain0): 172750013 irq321: t6nex0:1ae:325 @cpu4(domain0): 173559485 irq322: t6nex0:1af:327 @cpu5(domain0): 227842473 irq323: t5nex0:err:329 @cpu0(domain1): 0 irq324: t5nex0:evt:331 @cpu0(domain1): 8 irq325: t5nex0:0a0:333 @cpu10(domain1): 1340449 irq326: t5nex0:0a1:335 @cpu11(domain1): 1128580 irq327: t5nex0:0a2:337 @cpu12(domain1): 1311599 irq328: t5nex0:0a3:339 @cpu13(domain1): 1157356 irq329: t5nex0:0a4:341 @cpu14(domain1): 1257426 irq330: t5nex0:0a5:343 @cpu15(domain1): 1169697 irq331: t5nex0:0a6:345 @cpu16(domain1): 1089689 irq332: t5nex0:0a7:347 @cpu17(domain1): 1117782 irq333: t5nex0:0a8:349 @cpu18(domain1): 1186770 irq334: t5nex0:0a9:351 @cpu19(domain1): 1147015 irq335: t5nex0:0aa:353 @cpu10(domain1): 1238148 irq336: t5nex0:0ab:355 @cpu11(domain1): 1134259 irq337: t5nex0:0ac:357 @cpu12(domain1): 1262301 irq338: t5nex0:0ad:359 @cpu13(domain1): 1233933 irq339: t5nex0:0ae:361 @cpu14(domain1): 1284298 irq340: t5nex0:0af:363 @cpu15(domain1): 1257873 irq341: t5nex0:1a0:365 @cpu16(domain1): 204307929 irq342: t5nex0:1a1:367 @cpu17(domain1): 221035308 irq343: t5nex0:1a2:369 @cpu18(domain1):
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On Fri, Feb 14, 2020 at 6:25 PM Rudy wrote: > On 2/12/20 7:21 PM, Rudy wrote: > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load (and > high interrupts). > > > Looking better! I made some changes based on BSDRP which I hadn't known > about -- I think ifqmaxlen was the tunable I overlooked. > > # > > https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local > net.link.ifqmaxlen="16384" > > This net.link.ifqmaxlen was set to help in case of lagg usage: I was not aware it could improve your use case. >From your first post, it looks like your setup is a 2 packages, 10 cores, 20 threads (disabled). And you have configured your Chelsio to use 16 queues (hw.cxgbe.Xrx=16): It's a good think to have a power of 2 number of queues with Chelsio, but I'm not sure it's a good idea to spread those queue across the 2 packages. So perhaps you should try: 1. To reduce queues to 8 queues and bind them to the local domain 2. Or keeping 16 queues, but re-enabling HyperThreading and bing them to the local domain too. (on -head with recent CPU and machdep.hyperthreading_intr_allowed, using hyper-threading improve forwarding performance). But anyway even with 16 queues spread over 2 domains, you should have better performance: https://github.com/ocochard/netbenches/blob/master/Xeon_E5-2650v4_2x12Cores-Chelsio_T520-CR/hw.cxgbe.nXxq/results/fbsd12-stable.r354440.BSDRP.1.96/README.md Notice that I never monitoring the CPU load during my benches. Increasing the hw.cxgbe.holdoff_timer_idx was a good idea: I would expect lower interrupt usage too. Did you monitor the QPI link usage ? (kldload cpuctl && pcm-numa.x) ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 2/12/20 7:21 PM, Rudy wrote: > I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Looking better! I made some changes based on BSDRP which I hadn't known about -- I think ifqmaxlen was the tunable I overlooked. # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" Also, I ran chelsio_affinity to bind queues to specific CPU cores. The script only supports a single t5 card, I am revising and will submit a patch that will do multiple t5 and t6 cards. I made both changes at once, and rebooted, so we'll never know which fixed it. ;) Right now, I have: #/boot/loader.conf # # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 #hw.cxgbe.pause_settings="0" # https://github.com/ocochard/BSDRP/blob/master/BSDRP/Files/boot/loader.conf.local net.link.ifqmaxlen="16384" #/etc/sysctl.conf # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (doesn't work with Cogent) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1 # drop any UDP packets to closed ports # hw.intr_storm_threshold=9000 net.inet.tcp.tso=0 ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 13.02.2020 06:21, Rudy wrote: > > > I'm having issues with a box that is acting as a BGP router for my > network. 3 Chelsio cards, two T5 and one T6. It was working great > until I turned up our first port on the T6. It seems like traffic > passing in from a T5 card and out the T6 causes a really high load (and > high interrupts). > > Traffic (not that much, right?) > > Dev RX bps TX bps RX PPS TX PPS Error > cc0 0 0 0 0 0 > cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink, > filtering inbound routes to keep TX low) > cxl0 287 k 2015 M 353 244 k 0 (our network) > cxl1 940 M 3115 M 176 k 360 k 0 (our network) > cxl2 634 M 1014 M 103 k 128 k 0 (our network) > cxl3 1 k 16 M 1 4 k 0 > cxl4 0 0 0 0 0 > cxl5 0 0 0 0 0 > cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0) > cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0) > ixl0 913 k 18 M 0 0 0 > ixl1 1 M 30 M 0 0 0 > lagg0 4019 M 1554 M 491 k 271 k 0 > lagg1 1 M 48 M 0 0 0 > FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg > 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 Hi, did you try to use pmcstat to determine what is the heaviest task for your system? # kldload hwpmc # pmcstat -S inst_retired.any -Tw1 Then capture several first lines from the output and quit using 'q'. Do you use some firewall? Also, can you show the snapshot from the `top -HPSIzts1` output. -- WBR, Andrey V. Elsukov signature.asc Description: OpenPGP digital signature
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
On 2/12/20 7:21 PM, Rudy wrote: I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Update, I have found the chelsio_affinity script and run that, doesn't seem to help. I did notice the T6 card is on numa-domain 0 and the T5 cards on on numa-domain 1. Does that matter? Here is the relevant part of dmesg: t6nex0: mem 0xc530-0xc537,0xc400-0xc4ff,0xc5984000-0xc5985fff irq 48 at device 0.4 numa-domain 0 on pci12 cc0: numa-domain 0 on t6nex0 cc1: numa-domain 0 on t6nex0 t6nex0: PCIe gen3 x16, 2 ports, 34 MSI-X interrupts, 66 eq, 33 iq t5nex0: mem 0xed30-0xed37,0xec00-0xecff,0xed984000-0xed985fff irq 88 at device 0.4 numa-domain 1 on pci18 cxl0: numa-domain 1 on t5nex0 cxl1: numa-domain 1 on t5nex0 cxl2: numa-domain 1 on t5nex0 cxl3: numa-domain 1 on t5nex0 t5nex0: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq t5nex1: mem 0xfb30-0xfb37,0xfa00-0xfaff,0xfb984000-0xfb985fff irq 98 at device 0.4 numa-domain 1 on pci20 cxl4: numa-domain 1 on t5nex1 cxl5: numa-domain 1 on t5nex1 cxl6: numa-domain 1 on t5nex1 cxl7: numa-domain 1 on t5nex1 t5nex1: PCIe gen3 x8, 4 ports, 66 MSI-X interrupts, 132 eq, 65 iq Box does have 20 cores and hyper-threading is disabled. sysctl hw.ncpu hw.ncpu: 20 I've added updated /boot/loader.conf to the following: hw.cxgbe.nrxq=16 hw.cxgbe.ntxq=16 hw.cxgbe.qsize_rxq=4096 hw.cxgbe.qsize_txq=4096 hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" hw.cxgbe.holdoff_timer_idx=3 machdep.hyperthreading_allowed="0" Box works fine unless I turn on TX on the cc1 device (I am filtering routes learned from that interface via BGP). Rudy ___ freebsd-net@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-net To unsubscribe, send any mail to "freebsd-net-unsubscr...@freebsd.org"
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
Load seems to be increasing even though outbound traffic is blocked. I'm seeing 'Discards In' rise up to about 4k/second when doing 4Gbps inbound traffic. I see the tx_pause counter incrementing... dev.cc.1.stats.tx_pause: 2911586 (~ 20/second increase) Any tips would be great, I'm stumped. 20 cores seems like plenty to do much higher PPS based on what I see on the freebsd-net list. Thanks for any pointers, Rudy On 2/12/20 7:21 PM, Rudy wrote: I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Attached graphs (will that work on a mailing list?) shows the load going up. The work around at this point is that I block inbound routes (out bound traffic) on cc1. I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't seem to lower Interrupts. # netstat -rn | wc -l 891149 CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle Traffic (not that much, right?) Dev RX bps TX bps RX PPS TX PPS Error cc0 0 0 0 0 0 cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink, filtering inbound routes to keep TX low) cxl0 287 k 2015 M 353 244 k 0 (our network) cxl1 940 M 3115 M 176 k 360 k 0 (our network) cxl2 634 M 1014 M 103 k 128 k 0 (our network) cxl3 1 k 16 M 1 4 k 0 cxl4 0 0 0 0 0 cxl5 0 0 0 0 0 cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0) cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0) ixl0 913 k 18 M 0 0 0 ixl1 1 M 30 M 0 0 0 lagg0 4019 M 1554 M 491 k 271 k 0 lagg1 1 M 48 M 0 0 0 FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 # /boot/loader.conf # ZFS! kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, 40/100Gbps prots) devices # built into kernel... # if_cxgbe_load="YES" # t5fw_cfg_load="YES" # t6fw_cfg_load="YES" # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" # /etc/sysctl.conf net.inet6.ip6.redirect=0 net.inet.ip.redirect=0 # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (no link to Cogent otherwise) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 # TODO: try this, https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html # (Default was 2) hw.cxgbe.nm_holdoff_tmr_idx=3 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1 # drop any UDP packets to closed ports # vmstat -i interrupt total rate cpu0:timer 54113958 1102 cpu1:timer 55286089 1125 cpu2:timer 53479994 1089 cpu3:timer 55210439 1124 cpu4:timer 25989916 529 cpu5:timer 25908493 527 cpu6:timer 55187512 1123 cpu7:timer 54097494 1101 cpu8:timer 54111676 1101 cpu9:timer 54541125 1110 cpu10:timer 55283621 1125 cpu11:timer 55285466 1125 cpu12:timer 55283322 1125 cpu13:timer 55287200 1125 cpu14:timer 55293560 1126 cpu15:timer 55286002 1125 cpu16:timer 55285700 1125 cpu17:timer 55288330 1125 cpu18:timer 55292101 1126 cpu19:timer 55287826 1125 irq265: xhci0 85784 2 irq266: ahci1 720421 15 irq267: ixl0:aq 1 0 irq268: ixl0:rxq0 8568015 174 irq269: ixl0:rxq1
Re: Issue with BGP router / high interrupt / Chelsio / FreeBSD 12.1
I'm having issues with a box that is acting as a BGP router for my network. 3 Chelsio cards, two T5 and one T6. It was working great until I turned up our first port on the T6. It seems like traffic passing in from a T5 card and out the T6 causes a really high load (and high interrupts). Attached graphs (will that work on a mailing list?) shows the load going up. The work around at this point is that I block inbound routes (out bound traffic) on cc1. I tried setting hw.cxgbe.nm_holdoff_tmr_idx to 3 and 5, but it didn't seem to lower Interrupts. # netstat -rn | wc -l 891149 CPU: 0.2% user, 0.0% nice, 0.6% system, 64.4% interrupt, 34.8% idle Traffic (not that much, right?) Dev RX bps TX bps RX PPS TX PPS Error cc0 0 0 0 0 0 cc1 2212 M 7 M 250 k 6 k 0 (100Gbps uplink, filtering inbound routes to keep TX low) cxl0 287 k 2015 M 353 244 k 0 (our network) cxl1 940 M 3115 M 176 k 360 k 0 (our network) cxl2 634 M 1014 M 103 k 128 k 0 (our network) cxl3 1 k 16 M 1 4 k 0 cxl4 0 0 0 0 0 cxl5 0 0 0 0 0 cxl6 2343 M 791 M 275 k 137 k 0 (IX , part of lagg0) cxl7 1675 M 762 M 215 k 133 k 0 (IX , part of lagg0) ixl0 913 k 18 M 0 0 0 ixl1 1 M 30 M 0 0 0 lagg0 4019 M 1554 M 491 k 271 k 0 lagg1 1 M 48 M 0 0 0 FreeBSD 12.1-STABLE orange 976 Bytes/Packetavg 1:42PM up 13:25, 5 users, load averages: 9.38, 10.43, 9.827 # /boot/loader.conf # ZFS! kern.geom.label.disk_ident.enable="0" kern.geom.label.gptid.enable="0" zfs_load="YES" # Chelsio drivers for cxl (T5 card, 10Gbps ports) and cc (T6 card, 40/100Gbps prots) devices # built into kernel... # if_cxgbe_load="YES" # t5fw_cfg_load="YES" # t6fw_cfg_load="YES" # Before FreeBSD 13, threading bad on router: https://calomel.org/freebsd_network_tuning.html machdep.hyperthreading_allowed="0" # https://wiki.freebsd.org/10gFreeBSD/Router hw.cxgbe.toecaps_allowed="0" hw.cxgbe.rdmacaps_allowed="0" hw.cxgbe.iscsicaps_allowed="0" hw.cxgbe.fcoecaps_allowed="0" # /etc/sysctl.conf net.inet6.ip6.redirect=0 net.inet.ip.redirect=0 # FRR needs big buffers for OSPF kern.ipc.maxsockbuf=16777216 # Turn FEC off (no link to Cogent otherwise) dev.cc.0.fec=0 dev.cc.1.fec=0 # Entropy not from LAN ports... slows them down. kern.random.harvest.mask=65551 # TODO: try this, https://lists.freebsd.org/pipermail/freebsd-net/2020-February/055299.html # (Default was 2) hw.cxgbe.nm_holdoff_tmr_idx=3 net.inet.icmp.icmplim=400 net.inet.icmp.maskrepl=0 net.inet.icmp.log_redirect=0 net.inet.icmp.drop_redirect=1 net.inet.tcp.drop_synfin=1 net.inet.tcp.blackhole=2 # drop any TCP packets to closed ports net.inet.tcp.msl=7500 # close lost tcp connections in 7.5 seconds (default 30) net.inet.udp.blackhole=1 # drop any UDP packets to closed ports # vmstat -i interrupt total rate cpu0:timer 54113958 1102 cpu1:timer 55286089 1125 cpu2:timer 53479994 1089 cpu3:timer 55210439 1124 cpu4:timer 25989916 529 cpu5:timer 25908493 527 cpu6:timer 55187512 1123 cpu7:timer 54097494 1101 cpu8:timer 54111676 1101 cpu9:timer 54541125 1110 cpu10:timer 55283621 1125 cpu11:timer 55285466 1125 cpu12:timer 55283322 1125 cpu13:timer 55287200 1125 cpu14:timer 55293560 1126 cpu15:timer 55286002 1125 cpu16:timer 55285700 1125 cpu17:timer 55288330 1125 cpu18:timer 55292101 1126 cpu19:timer 55287826 1125 irq265: xhci0 85784 2 irq266: ahci1 720421 15 irq267: ixl0:aq 1 0 irq268: ixl0:rxq0 8568015 174 irq269: ixl0:rxq1 10726322 218 irq270: ixl0:rxq2 7915180 161 irq271: ixl0:rxq3 8527106 174 irq272: ixl0:rxq4 13080418 266 irq273: ixl0:rxq5 7456434 152 irq274: ixl0:rxq6 8012988 163 irq275: ixl0:rxq7 8807168 179 irq276: ixl0:rxq8 9247761 188 irq277: ixl0:rxq9 8439429 172 irq278: ixl1:aq
