Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Hi,



So basically this is my conclusion, if I cpuset iperf on at least the receiving 
end i get great performance. Anything outside of that is random. I've tried 
just about every network tuning knob in FreeBSD as well as what Mellanox 
recommends in their driver manual, none of these make any significant impact, 
sometimes they even reduce performance. So I m hoping when this goes into 
production the scheduler will be sane enough to do the same, but we shall see.



Thanks.






 On Fri, 17 Jun 2022 11:03:04 -0400 Dave Cottlehuber  
wrote ---



On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote: 
> Hi, 
> 
> I believe you hit the nail on the head! I am now getting consistent 
> high speeds, even higher than on Linux! Is this a problem with the 
> scheduler? Should someone in that area of expertise be made aware of 
> this? More importantly i guess, would this affect real world 
> performance, these servers will be running RabbitMQ (it uses quite a 
> bit of bandwidth) and PostgreSQL w/ replication. 
 
pinning cores for unimpeded access is very common for high performance systems. 
Do this both for the nics and also your apps. Be mindful of the NUMA topooogy. 
 
You should look into both the  erlang scheduler flags for core pinning, and 
also ensuring that your erlang processes have unimpeded access to their own 
cores too. A reasonable approach is to make a simple cowboy or Phoenix app and 
hammer it with wrk or similar load tool to get a feel for things, and then 
profile and tune your own app based on those specific results. 
 
For rabbit there is an excellent load testing tool from the pivotal team if you 
don’t have suitable load generators yourselves. 
 
Tsung is an excellent tool if you put in the work to craft something specific 
for your use case. 
 
Please post back to the list with your specific findings and nic/ tcp tunables, 
these are very helpful for the next person! 
 
Dave 
 





Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use of 
the addressee(s) and may contain information that is privileged or 
confidential. If you are not the intended recipient, or responsible for 
delivering the information to the intended recipient, you are hereby notified 
that any dissemination, distribution, printing or copying of this e-mail and 
any attachments is strictly prohibited. If this e-mail and any attachments were 
received in error, please notify the sender by reply e-mail and delete the 
original message.

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Dave Cottlehuber]

On Fri, 17 Jun 2022, at 02:38, Mike Jakubik wrote:
> Hi,
>
> I believe you hit the nail on the head! I am now getting consistent 
> high speeds, even higher than on Linux! Is this a problem with the 
> scheduler? Should someone in that area of expertise be made aware of 
> this? More importantly i guess, would this affect real world 
> performance, these servers will be running RabbitMQ (it uses quite a 
> bit of bandwidth) and PostgreSQL w/ replication.

pinning cores for unimpeded access is very common for high performance systems. 
Do this both for the nics and also your apps. Be mindful of the NUMA topooogy.

You should look into both the  erlang scheduler flags for core pinning, and 
also ensuring that your erlang processes have unimpeded access to their own 
cores too. A reasonable approach is to make a simple cowboy or Phoenix app and 
hammer it with wrk or similar load tool to get a feel for things, and then 
profile and tune your own app based on those specific results.

For rabbit there is an excellent load testing tool from the pivotal team if you 
don’t have suitable load generators yourselves.

Tsung is an excellent tool if you put in the work to craft something specific 
for your use case.

Please post back to the list with your specific findings and nic/ tcp tunables, 
these are very helpful for the next person!

Dave 

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Hi,



I believe you hit the nail on the head! I am now getting consistent high 
speeds, even higher than on Linux! Is this a problem with the scheduler? Should 
someone in that area of expertise be made aware of this? More importantly i 
guess, would this affect real world performance, these servers will be running 
RabbitMQ (it uses quite a bit of bandwidth) and PostgreSQL w/ replication.



[root@db-01 ~]# cpuset -l 1 iperf3 -s 

---

Server listening on 5201 (test #1)

---



[root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 22445 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  2.17 GBytes  18.7 Gbits/sec  263   1.13 MBytes

[  5]   1.00-2.00   sec  2.11 GBytes  18.1 Gbits/sec    0   1.14 MBytes

[  5]   2.00-3.00   sec  2.46 GBytes  21.2 Gbits/sec  696    893 KBytes

[  5]   3.00-4.00   sec  2.49 GBytes  21.4 Gbits/sec  857    447 KBytes

[  5]   4.00-5.00   sec  2.36 GBytes  20.3 Gbits/sec  281    566 KBytes

[  5]   5.00-6.00   sec  2.36 GBytes  20.3 Gbits/sec  348    593 KBytes

[  5]   6.00-7.00   sec  2.28 GBytes  19.6 Gbits/sec    0    694 KBytes

[  5]   7.00-8.00   sec  2.20 GBytes  18.9 Gbits/sec  165    611 KBytes

[  5]   8.00-9.00   sec  2.26 GBytes  19.5 Gbits/sec  183    541 KBytes

[  5]   9.00-10.00  sec  2.26 GBytes  19.4 Gbits/sec    0    625 KBytes

[  5]  10.00-11.00  sec  2.27 GBytes  19.5 Gbits/sec  165    608 KBytes

[  5]  11.00-12.00  sec  2.22 GBytes  19.0 Gbits/sec    0    679 KBytes

[  5]  12.00-13.00  sec  2.22 GBytes  19.0 Gbits/sec    0    726 KBytes

[  5]  13.00-14.00  sec  2.25 GBytes  19.3 Gbits/sec  174    610 KBytes

[  5]  14.00-15.00  sec  2.19 GBytes  18.8 Gbits/sec    0    644 KBytes

[  5]  15.00-16.00  sec  2.20 GBytes  18.9 Gbits/sec    0    704 KBytes

[  5]  16.00-17.00  sec  2.18 GBytes  18.7 Gbits/sec    0    784 KBytes

[  5]  17.00-18.00  sec  2.18 GBytes  18.7 Gbits/sec    0    820 KBytes

[  5]  18.00-19.00  sec  2.19 GBytes  18.8 Gbits/sec    0    874 KBytes

[  5]  19.00-20.00  sec  2.19 GBytes  18.8 Gbits/sec    0    902 KBytes

[  5]  20.00-21.00  sec  2.23 GBytes  19.1 Gbits/sec  202    661 KBytes

[  5]  21.00-22.00  sec  2.30 GBytes  19.8 Gbits/sec  171    894 KBytes

[  5]  22.00-23.00  sec  2.33 GBytes  20.0 Gbits/sec  341    409 KBytes

[  5]  23.00-24.00  sec  2.29 GBytes  19.7 Gbits/sec  167    594 KBytes

[  5]  24.00-25.00  sec  2.28 GBytes  19.6 Gbits/sec    0    714 KBytes

[  5]  25.00-26.00  sec  2.29 GBytes  19.7 Gbits/sec  166    604 KBytes

[  5]  26.00-27.00  sec  2.26 GBytes  19.4 Gbits/sec    0    705 KBytes

[  5]  27.00-28.00  sec  1.86 GBytes  16.0 Gbits/sec    0    763 KBytes

[  5]  28.00-29.00  sec  2.25 GBytes  19.3 Gbits/sec    0    816 KBytes

[  5]  29.00-30.00  sec  2.24 GBytes  19.2 Gbits/sec    0    869 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  67.4 GBytes  19.3 Gbits/sec  4179 sender

[  5]   0.00-30.00  sec  67.4 GBytes  19.3 Gbits/sec  receiver



iperf Done.

[root@db-02 ~]# cpuset -l 1 iperf3 -i 1 -t 30 -c db-01

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 51998 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  1.99 GBytes  17.1 Gbits/sec  276    456 KBytes

[  5]   1.00-2.00   sec  2.00 GBytes  17.2 Gbits/sec    0    593 KBytes

[  5]   2.00-3.00   sec  2.27 GBytes  19.5 Gbits/sec  166    556 KBytes

[  5]   3.00-4.00   sec  2.31 GBytes  19.8 Gbits/sec  186    558 KBytes

[  5]   4.00-5.00   sec  2.31 GBytes  19.8 Gbits/sec  168    558 KBytes

[  5]   5.00-6.00   sec  2.29 GBytes  19.6 Gbits/sec  182    545 KBytes

[  5]   6.00-7.00   sec  2.29 GBytes  19.7 Gbits/sec  171    507 KBytes

[  5]   7.00-8.00   sec  2.29 GBytes  19.7 Gbits/sec  180    517 KBytes

[  5]   8.00-9.00   sec  2.30 GBytes  19.7 Gbits/sec  174    568 KBytes

[  5]   9.00-10.00  sec  2.28 GBytes  19.6 Gbits/sec  178    434 KBytes

[  5]  10.00-11.00  sec  2.26 GBytes  19.4 Gbits/sec    0    583 KBytes

[  5]  11.00-12.00  sec  2.31 GBytes  19.8 Gbits/sec  180    443 KBytes

[  5]  12.00-13.00  sec  2.32 GBytes  19.9 Gbits/sec  166    564 KBytes

[  5]  13.00-14.00  sec  2.35 GBytes  20.2 Gbits/sec  350    424 KBytes

[  5]  14.00-15.02  sec  1.49 GBytes  12.5 Gbits/sec  175   1.41 KBytes

[  5]  15.02-16.00  sec  1.73 GBytes  15.2 Gbits/sec    0   1.12 MBytes

[  5]  16.00-17.00  sec  2.22 GBytes  19.1 Gbits/sec  171    784 KBytes

[  5]  17.00-18.00  sec  2.34 GBytes  20.1 Gbits/sec  348    436 KBytes

[  5]  18.00-19.00  sec  2.34 GBytes  20.1 Gbits/sec  350    547 KBytes

[  5]  19.00-20.00  sec  2.31 GBytes  19.9 Gbits/sec  173    595 

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Alexander V. Chernikov]

> On 16 Jun 2022, at 21:48, Mike Jakubik  
> wrote:
> 
> After multiple tests and tweaks i believe the issue is not with the HW or 
> Numa related (Infinity fabric should do around 32GB) but rather with FreeBSD 
> TCP/IP stack. It's like it cant figure itself out properly for the speed that 
> the HW can do, i keep getting widely varying results when testing. Below is 
> an example of two tests, with about a 15 second break in between the two.
Does pinning iperf to a specific CPU core (or range) address the variety part? 
e.g. cpuset -l 1 perf …
The output you shared above shows CPU#83 as the core iperf is running on. Just 
wondering whether the scheduler migrates iperf too often, thrashing the caches.

> 
> [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 
> Connecting to host db-01, port 5201
> [  5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5201
> [ ID] Interval   Transfer Bitrate Retr  Cwnd
> [  5]   0.00-1.00   sec   991 MBytes  8.32 Gbits/sec  268579 KBytes
> [  5]   1.00-2.00   sec   945 MBytes  7.93 Gbits/sec  369777 KBytes
> [  5]   2.00-3.00   sec   793 MBytes  6.65 Gbits/sec   60   1.03 MBytes
> [  5]   3.00-4.00   sec   666 MBytes  5.59 Gbits/sec  203976 KBytes
> [  5]   4.00-5.01   sec   575 MBytes  4.78 Gbits/sec  202   1.13 MBytes
> [  5]   5.01-6.00   sec   169 MBytes  1.43 Gbits/sec  134699 KBytes
> [  5]   6.00-7.00   sec  1.21 GBytes  10.4 Gbits/sec  383   1.08 MBytes
> [  5]   7.00-8.00   sec  1.21 GBytes  10.4 Gbits/sec0   1.16 MBytes
> [  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec  124780 KBytes
> [  5]   9.00-10.00  sec   690 MBytes  5.79 Gbits/sec  316605 KBytes
> [  5]  10.00-11.00  sec   685 MBytes  5.75 Gbits/sec   97854 KBytes
> [  5]  11.00-12.00  sec  1.08 GBytes  9.30 Gbits/sec  383538 KBytes
> [  5]  12.00-13.00  sec   682 MBytes  5.72 Gbits/sec   88870 KBytes
> [  5]  13.00-14.00  sec   678 MBytes  5.69 Gbits/sec  123964 KBytes
> [  5]  14.00-15.00  sec   670 MBytes  5.62 Gbits/sec  290763 KBytes
> [  5]  15.00-16.00  sec  1.01 GBytes  8.71 Gbits/sec  228   1.08 MBytes
> [  5]  16.00-17.00  sec   886 MBytes  7.44 Gbits/sec  118615 KBytes
> [  5]  17.00-18.00  sec   734 MBytes  6.16 Gbits/sec  291902 KBytes
> [  5]  18.00-19.00  sec  1.04 GBytes  8.96 Gbits/sec  212323 KBytes
> [  5]  19.00-20.00  sec   710 MBytes  5.96 Gbits/sec  193547 KBytes
> [  5]  20.00-21.00  sec   693 MBytes  5.82 Gbits/sec  370942 KBytes
> [  5]  21.00-22.00  sec   704 MBytes  5.91 Gbits/sec   80   1022 KBytes
> [  5]  22.00-23.00  sec  1.26 GBytes  10.8 Gbits/sec  262965 KBytes
> [  5]  23.00-24.00  sec   828 MBytes  6.94 Gbits/sec  202763 KBytes
> [  5]  24.00-25.00  sec   774 MBytes  6.49 Gbits/sec  227581 KBytes
> [  5]  25.00-26.00  sec   734 MBytes  6.15 Gbits/sec  256664 KBytes
> [  5]  26.00-27.00  sec   753 MBytes  6.32 Gbits/sec  331540 KBytes
> [  5]  27.00-28.00  sec   764 MBytes  6.41 Gbits/sec  298823 KBytes
> [  5]  28.00-29.00  sec   757 MBytes  6.35 Gbits/sec  123850 KBytes
> [  5]  29.00-30.00  sec   754 MBytes  6.32 Gbits/sec   74970 KBytes
> - - - - - - - - - - - - - - - - - - - - - - - - -
> [ ID] Interval   Transfer Bitrate Retr
> [  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec  6305 sender
> [  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec  receiver
> 
> iperf Done.
> 
> [root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01
> Connecting to host db-01, port 5201
> [  5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5201
> [ ID] Interval   Transfer Bitrate Retr  Cwnd
> [  5]   0.00-1.00   sec  1.81 GBytes  15.5 Gbits/sec0   1.11 MBytes
> [  5]   1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec0   1.11 MBytes
> [  5]   2.00-3.00   sec  1.98 GBytes  17.0 Gbits/sec0   1.11 MBytes
> [  5]   3.00-4.00   sec  2.11 GBytes  18.1 Gbits/sec0   1.11 MBytes
> [  5]   4.00-5.00   sec  2.12 GBytes  18.2 Gbits/sec0   1.11 MBytes
> [  5]   5.00-6.00   sec  2.16 GBytes  18.5 Gbits/sec0   1.11 MBytes
> [  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec0   1.12 MBytes
> [  5]   7.00-8.02   sec  1.28 GBytes  10.8 Gbits/sec0   1.12 MBytes
> [  5]   8.02-9.00   sec  1.83 GBytes  16.0 Gbits/sec0   1.17 MBytes
> [  5]   9.00-10.00  sec  1.91 GBytes  16.4 Gbits/sec0   1.20 MBytes
> [  5]  10.00-11.00  sec  1.79 GBytes  15.3 Gbits/sec0   1.60 MBytes
> [  5]  11.00-12.00  sec  1.77 GBytes  15.2 Gbits/sec0   1.60 MBytes
> [  5]  12.00-13.00  sec  1.69 GBytes  14.5 Gbits/sec0   1.61 MBytes
> [  5]  13.00-14.00  sec  1.57 GBytes  13.5 Gbits/sec0   1.61 MBytes
> [  5]  14.00-15.00  sec  1.60 GBytes  13.8 Gbits/sec0   1.61 MBytes
> [  5]  15.00-16.00  sec  1.89 GBytes  16.2 Gbits/sec0   1.61 MBytes
> [  5]  16.00-17.00  sec  1.76 GBytes  15.1 Gbits/sec0   1.61 MBytes
> [  5]  17.00-18.00  sec  1.93 GBytes  16.6 Gbits/sec 

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

After multiple tests and tweaks i believe the issue is not with the HW or Numa 
related (Infinity fabric should do around 32GB) but rather with FreeBSD TCP/IP 
stack. It's like it cant figure itself out properly for the speed that the HW 
can do, i keep getting widely varying results when testing. Below is an example 
of two tests, with about a 15 second break in between the two.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 49155 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec   991 MBytes  8.32 Gbits/sec  268    579 KBytes

[  5]   1.00-2.00   sec   945 MBytes  7.93 Gbits/sec  369    777 KBytes

[  5]   2.00-3.00   sec   793 MBytes  6.65 Gbits/sec   60   1.03 MBytes

[  5]   3.00-4.00   sec   666 MBytes  5.59 Gbits/sec  203    976 KBytes

[  5]   4.00-5.01   sec   575 MBytes  4.78 Gbits/sec  202   1.13 MBytes

[  5]   5.01-6.00   sec   169 MBytes  1.43 Gbits/sec  134    699 KBytes

[  5]   6.00-7.00   sec  1.21 GBytes  10.4 Gbits/sec  383   1.08 MBytes

[  5]   7.00-8.00   sec  1.21 GBytes  10.4 Gbits/sec    0   1.16 MBytes

[  5]   8.00-9.00   sec  1.32 GBytes  11.3 Gbits/sec  124    780 KBytes

[  5]   9.00-10.00  sec   690 MBytes  5.79 Gbits/sec  316    605 KBytes

[  5]  10.00-11.00  sec   685 MBytes  5.75 Gbits/sec   97    854 KBytes

[  5]  11.00-12.00  sec  1.08 GBytes  9.30 Gbits/sec  383    538 KBytes

[  5]  12.00-13.00  sec   682 MBytes  5.72 Gbits/sec   88    870 KBytes

[  5]  13.00-14.00  sec   678 MBytes  5.69 Gbits/sec  123    964 KBytes

[  5]  14.00-15.00  sec   670 MBytes  5.62 Gbits/sec  290    763 KBytes

[  5]  15.00-16.00  sec  1.01 GBytes  8.71 Gbits/sec  228   1.08 MBytes

[  5]  16.00-17.00  sec   886 MBytes  7.44 Gbits/sec  118    615 KBytes

[  5]  17.00-18.00  sec   734 MBytes  6.16 Gbits/sec  291    902 KBytes

[  5]  18.00-19.00  sec  1.04 GBytes  8.96 Gbits/sec  212    323 KBytes

[  5]  19.00-20.00  sec   710 MBytes  5.96 Gbits/sec  193    547 KBytes

[  5]  20.00-21.00  sec   693 MBytes  5.82 Gbits/sec  370    942 KBytes

[  5]  21.00-22.00  sec   704 MBytes  5.91 Gbits/sec   80   1022 KBytes

[  5]  22.00-23.00  sec  1.26 GBytes  10.8 Gbits/sec  262    965 KBytes

[  5]  23.00-24.00  sec   828 MBytes  6.94 Gbits/sec  202    763 KBytes

[  5]  24.00-25.00  sec   774 MBytes  6.49 Gbits/sec  227    581 KBytes

[  5]  25.00-26.00  sec   734 MBytes  6.15 Gbits/sec  256    664 KBytes

[  5]  26.00-27.00  sec   753 MBytes  6.32 Gbits/sec  331    540 KBytes

[  5]  27.00-28.00  sec   764 MBytes  6.41 Gbits/sec  298    823 KBytes

[  5]  28.00-29.00  sec   757 MBytes  6.35 Gbits/sec  123    850 KBytes

[  5]  29.00-30.00  sec   754 MBytes  6.32 Gbits/sec   74    970 KBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec  6305 sender

[  5]   0.00-30.00  sec  24.4 GBytes  6.98 Gbits/sec  receiver



iperf Done.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 25061 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  1.81 GBytes  15.5 Gbits/sec    0   1.11 MBytes

[  5]   1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec    0   1.11 MBytes

[  5]   2.00-3.00   sec  1.98 GBytes  17.0 Gbits/sec    0   1.11 MBytes

[  5]   3.00-4.00   sec  2.11 GBytes  18.1 Gbits/sec    0   1.11 MBytes

[  5]   4.00-5.00   sec  2.12 GBytes  18.2 Gbits/sec    0   1.11 MBytes

[  5]   5.00-6.00   sec  2.16 GBytes  18.5 Gbits/sec    0   1.11 MBytes

[  5]   6.00-7.00   sec  1.90 GBytes  16.3 Gbits/sec    0   1.12 MBytes

[  5]   7.00-8.02   sec  1.28 GBytes  10.8 Gbits/sec    0   1.12 MBytes

[  5]   8.02-9.00   sec  1.83 GBytes  16.0 Gbits/sec    0   1.17 MBytes

[  5]   9.00-10.00  sec  1.91 GBytes  16.4 Gbits/sec    0   1.20 MBytes

[  5]  10.00-11.00  sec  1.79 GBytes  15.3 Gbits/sec    0   1.60 MBytes

[  5]  11.00-12.00  sec  1.77 GBytes  15.2 Gbits/sec    0   1.60 MBytes

[  5]  12.00-13.00  sec  1.69 GBytes  14.5 Gbits/sec    0   1.61 MBytes

[  5]  13.00-14.00  sec  1.57 GBytes  13.5 Gbits/sec    0   1.61 MBytes

[  5]  14.00-15.00  sec  1.60 GBytes  13.8 Gbits/sec    0   1.61 MBytes

[  5]  15.00-16.00  sec  1.89 GBytes  16.2 Gbits/sec    0   1.61 MBytes

[  5]  16.00-17.00  sec  1.76 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  17.00-18.00  sec  1.93 GBytes  16.6 Gbits/sec    0   1.61 MBytes

[  5]  18.00-19.00  sec  1.77 GBytes  15.2 Gbits/sec    0   1.61 MBytes

[  5]  19.00-20.00  sec  1.68 GBytes  14.5 Gbits/sec    0   1.61 MBytes

[  5]  20.00-21.00  sec  1.66 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]  21.00-22.00  sec  1.75 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  22.00-23.00  sec  1.76 GBytes  15.1 Gbits/sec    0   1.61 MBytes

[  5]  

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Actually, i believe its the disabling to HW LRO that makes the difference (i 
disabled it and rx/tx pause previously). With rx/tx pause on and LRO off i get 
similar results. The throughput is still very sporadic though.



Connecting to host db-01, port 5201 

[  5] local 192.168.10.31 port 59055 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  1.83 GBytes  15.7 Gbits/sec    0   1.11 MBytes

[  5]   1.00-2.00   sec  1.83 GBytes  15.7 Gbits/sec    0   1.11 MBytes

[  5]   2.00-3.00   sec  2.06 GBytes  17.7 Gbits/sec    0   1.12 MBytes

[  5]   3.00-4.00   sec  2.15 GBytes  18.5 Gbits/sec    0   1.12 MBytes

[  5]   4.00-5.00   sec  2.15 GBytes  18.4 Gbits/sec    0   1.12 MBytes

[  5]   5.00-6.00   sec  2.15 GBytes  18.5 Gbits/sec    0   1.13 MBytes

[  5]   6.00-7.00   sec  2.15 GBytes  18.5 Gbits/sec    0   1.13 MBytes

[  5]   7.00-8.00   sec  2.15 GBytes  18.5 Gbits/sec    0   1.13 MBytes

[  5]   8.00-9.00   sec  2.15 GBytes  18.5 Gbits/sec    0   1.13 MBytes

[  5]   9.00-10.00  sec  2.15 GBytes  18.5 Gbits/sec    0   1.13 MBytes

[  5]  10.00-11.00  sec  1.72 GBytes  14.7 Gbits/sec    0   1.14 MBytes

[  5]  11.00-12.00  sec  1.18 GBytes  10.1 Gbits/sec  131    932 KBytes

[  5]  12.00-13.00  sec   739 MBytes  6.20 Gbits/sec  352    913 KBytes

[  5]  13.00-14.00  sec   724 MBytes  6.07 Gbits/sec  142    636 KBytes

[  5]  14.00-15.00  sec   759 MBytes  6.37 Gbits/sec  246    748 KBytes

[  5]  15.00-16.00  sec   748 MBytes  6.27 Gbits/sec   22    962 KBytes

[  5]  16.00-17.00  sec   753 MBytes  6.32 Gbits/sec  170    711 KBytes

[  5]  17.00-18.01  sec   690 MBytes  5.76 Gbits/sec  389    981 KBytes

[  5]  18.01-19.00  sec   771 MBytes  6.50 Gbits/sec  145    896 KBytes

[  5]  19.00-20.00  sec   741 MBytes  6.22 Gbits/sec   13    952 KBytes

[  5]  20.00-21.00  sec   743 MBytes  6.23 Gbits/sec   86    985 KBytes

[  5]  21.00-22.00  sec   751 MBytes  6.30 Gbits/sec  245    944 KBytes

[  5]  22.00-23.00  sec   724 MBytes  6.07 Gbits/sec  178    774 KBytes

[  5]  23.00-24.00  sec   728 MBytes  6.11 Gbits/sec  348    546 KBytes

[  5]  24.00-25.01  sec   622 MBytes  5.18 Gbits/sec  215    842 KBytes

[  5]  25.01-26.00  sec   920 MBytes  7.78 Gbits/sec  359   1.03 MBytes

[  5]  26.00-27.00  sec  1.02 GBytes  8.80 Gbits/sec  385   1011 KBytes

[  5]  27.00-28.00  sec   999 MBytes  8.38 Gbits/sec  161    757 KBytes

[  5]  28.00-29.00  sec  1.68 GBytes  14.4 Gbits/sec    0   1.03 MBytes

[  5]  29.00-30.00  sec  1.65 GBytes  14.2 Gbits/sec    0   1.51 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  39.2 GBytes  11.2 Gbits/sec  3587 sender

[  5]   0.00-30.00  sec  39.2 GBytes  11.2 Gbits/sec  receiver





[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 60931 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec   931 MBytes  7.81 Gbits/sec  146    689 KBytes

[  5]   1.00-2.00   sec   674 MBytes  5.66 Gbits/sec  267    575 KBytes

[  5]   2.00-3.00   sec   641 MBytes  5.38 Gbits/sec  270    485 KBytes

[  5]   3.00-4.00   sec   668 MBytes  5.60 Gbits/sec  221    801 KBytes

[  5]   4.00-5.01   sec   532 MBytes  4.40 Gbits/sec  208    981 KBytes

[  5]   5.01-6.00   sec   838 MBytes  7.13 Gbits/sec  291    949 KBytes

[  5]   6.00-7.00   sec   703 MBytes  5.90 Gbits/sec  178    628 KBytes

[  5]   7.00-8.00   sec   696 MBytes  5.84 Gbits/sec   85    928 KBytes

[  5]   8.00-9.00   sec  1.06 GBytes  9.14 Gbits/sec   66    565 KBytes

[  5]   9.00-10.00  sec   975 MBytes  8.18 Gbits/sec    0    801 KBytes

[  5]  10.00-11.00  sec   679 MBytes  5.70 Gbits/sec    0    859 KBytes

[  5]  11.00-12.01  sec   628 MBytes  5.23 Gbits/sec  123   1.12 MBytes

[  5]  12.01-13.00  sec   731 MBytes  6.16 Gbits/sec  311    864 KBytes

[  5]  13.00-14.00  sec   696 MBytes  5.86 Gbits/sec   39    798 KBytes

[  5]  14.00-15.00  sec   914 MBytes  7.66 Gbits/sec   65   1024 KBytes

[  5]  15.00-16.00  sec  1.66 GBytes  14.2 Gbits/sec    0   1.47 MBytes

[  5]  16.00-17.00  sec   955 MBytes  8.01 Gbits/sec  222    260 KBytes

[  5]  17.00-18.00  sec   769 MBytes  6.45 Gbits/sec    0    520 KBytes

[  5]  18.00-19.00  sec   781 MBytes  6.55 Gbits/sec  119    432 KBytes

[  5]  19.00-20.00  sec   779 MBytes  6.54 Gbits/sec  148    816 KBytes

[  5]  20.00-21.00  sec   771 MBytes  6.47 Gbits/sec  127    965 KBytes

[  5]  21.00-22.00  sec   745 MBytes  6.25 Gbits/sec  337    927 KBytes

[  5]  22.00-23.00  sec   637 MBytes  5.35 Gbits/sec    0    938 KBytes

[  5]  23.00-24.00  sec   626 MBytes  5.25 Gbits/sec    0    938 KBytes

[  5]  24.00-25.00  sec  1.96 GBytes  16.8 Gbits/sec  185    862 KBytes

[  5]  25.00-26.00  sec  1.71 GBytes  14.7 Gbits/sec   22    945 KBytes

[  5]  

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Disabling rx/tx pause seems to produce higher peaks.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 10146 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  1.89 GBytes  16.2 Gbits/sec    0   1.10 MBytes

[  5]   1.00-2.00   sec  1.86 GBytes  15.9 Gbits/sec    0   1.10 MBytes

[  5]   2.00-3.00   sec  2.05 GBytes  17.6 Gbits/sec    0   1.11 MBytes

[  5]   3.00-4.00   sec   859 MBytes  7.20 Gbits/sec   21    938 KBytes

[  5]   4.00-5.00   sec   652 MBytes  5.47 Gbits/sec    0   1.01 MBytes

[  5]   5.00-6.00   sec   659 MBytes  5.53 Gbits/sec    0   1.03 MBytes

[  5]   6.00-7.00   sec   666 MBytes  5.59 Gbits/sec    0   1.05 MBytes

[  5]   7.00-8.00   sec   657 MBytes  5.51 Gbits/sec   98    989 KBytes

[  5]   8.00-9.00   sec   665 MBytes  5.58 Gbits/sec  139    712 KBytes

[  5]   9.00-10.00  sec   647 MBytes  5.43 Gbits/sec    0   1.02 MBytes

[  5]  10.00-11.00  sec   650 MBytes  5.45 Gbits/sec    4    606 KBytes

[  5]  11.00-12.00  sec  1.53 GBytes  13.1 Gbits/sec  358   1.07 MBytes

[  5]  12.00-13.00  sec  2.10 GBytes  18.1 Gbits/sec  162    837 KBytes

[  5]  13.00-14.00  sec  2.09 GBytes  18.0 Gbits/sec  332    838 KBytes

[  5]  14.00-15.00  sec  2.43 GBytes  20.9 Gbits/sec  639    747 KBytes

[  5]  15.00-16.00  sec  2.38 GBytes  20.4 Gbits/sec  612   1.02 MBytes

[  5]  16.00-17.00  sec  2.25 GBytes  19.3 Gbits/sec  535   1.24 MBytes

[  5]  17.00-18.00  sec  2.52 GBytes  21.6 Gbits/sec  818    423 KBytes

[  5]  18.00-19.00  sec  2.29 GBytes  19.7 Gbits/sec  218    444 KBytes

[  5]  19.00-20.00  sec  2.29 GBytes  19.7 Gbits/sec  114    859 KBytes

[  5]  20.00-21.00  sec  1.65 GBytes  14.1 Gbits/sec  100    541 KBytes

[  5]  21.00-22.00  sec  1.01 GBytes  8.67 Gbits/sec    0    639 KBytes

[  5]  22.00-23.00  sec   625 MBytes  5.24 Gbits/sec    0    648 KBytes

[  5]  23.00-24.00  sec   630 MBytes  5.28 Gbits/sec    0    648 KBytes

[  5]  24.00-25.00  sec  1.56 GBytes  13.4 Gbits/sec    0    702 KBytes

[  5]  25.00-26.00  sec  1.78 GBytes  15.3 Gbits/sec  118    406 KBytes

[  5]  26.00-27.00  sec  1.37 GBytes  11.8 Gbits/sec  105    890 KBytes

[  5]  27.00-28.00  sec  1.82 GBytes  15.6 Gbits/sec  104    963 KBytes

[  5]  28.00-29.00  sec  1.68 GBytes  14.4 Gbits/sec    0   1.20 MBytes

[  5]  29.00-30.00  sec  1.67 GBytes  14.4 Gbits/sec    0   1.38 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - - 

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  44.8 GBytes  12.8 Gbits/sec  4477 sender

[  5]   0.00-30.01  sec  44.8 GBytes  12.8 Gbits/sec  receiver



After a few runs:



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 52152 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec  1.91 GBytes  16.4 Gbits/sec   67    606 KBytes

[  5]   1.00-2.00   sec  1.78 GBytes  15.3 Gbits/sec    0   1.07 MBytes

[  5]   2.00-3.00   sec  1.60 GBytes  13.7 Gbits/sec    0   1.54 MBytes

[  5]   3.00-4.00   sec  1.61 GBytes  13.8 Gbits/sec    0   1.61 MBytes

[  5]   4.00-5.00   sec  1.66 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]   5.00-6.00   sec  1.67 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]   6.00-7.00   sec  1.65 GBytes  14.1 Gbits/sec    0   1.61 MBytes

[  5]   7.00-8.00   sec  1.70 GBytes  14.6 Gbits/sec    0   1.61 MBytes

[  5]   8.00-9.00   sec  1.72 GBytes  14.8 Gbits/sec    0   1.61 MBytes

[  5]   9.00-10.00  sec  1.85 GBytes  15.9 Gbits/sec    0   1.61 MBytes

[  5]  10.00-11.00  sec  1.81 GBytes  15.5 Gbits/sec    0   1.61 MBytes

[  5]  11.00-12.00  sec  1.67 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]  12.00-13.00  sec  1.66 GBytes  14.3 Gbits/sec    0   1.61 MBytes

[  5]  13.00-14.00  sec  1.83 GBytes  15.7 Gbits/sec    0   1.61 MBytes

[  5]  14.00-15.00  sec  1.18 GBytes  10.1 Gbits/sec    0    794 KBytes

[  5]  15.00-16.00  sec  1.67 GBytes  14.4 Gbits/sec    0   1.60 MBytes

[  5]  16.00-17.00  sec  1.73 GBytes  14.8 Gbits/sec    0   1.60 MBytes

[  5]  17.00-18.00  sec  1.73 GBytes  14.9 Gbits/sec    0   1.60 MBytes

[  5]  18.00-19.00  sec  1.83 GBytes  15.7 Gbits/sec    0   1.61 MBytes

[  5]  19.00-20.00  sec  1.77 GBytes  15.2 Gbits/sec    0   1.61 MBytes

[  5]  20.00-21.00  sec  1.80 GBytes  15.5 Gbits/sec    0   1.61 MBytes

[  5]  21.00-22.00  sec  2.03 GBytes  17.4 Gbits/sec    0   1.61 MBytes

[  5]  22.00-23.00  sec  1.88 GBytes  16.1 Gbits/sec    0   1.61 MBytes

[  5]  23.00-24.00  sec  1.80 GBytes  15.5 Gbits/sec    0   1.61 MBytes

[  5]  24.00-25.01  sec  1.59 GBytes  13.6 Gbits/sec    0   1.61 MBytes

[  5]  25.01-26.00  sec  1.33 GBytes  11.4 Gbits/sec    0   1.61 MBytes

[  5]  26.00-27.00  sec  1.71 GBytes  14.7 Gbits/sec    0   1.61 MBytes

[  5]  27.00-28.00  sec  1.71 GBytes  

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Yes, it is the default of 1500. If I set it to 9000 I get some bizarre network 
behavior.





 On Tue, 14 Jun 2022 09:45:10 -0400 Andrey V. Elsukov 
 wrote 






Hi, 
 
Do you have the same MTU size on linux machine? 
 






Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use of 
the addressee(s) and may contain information that is privileged or 
confidential. If you are not the intended recipient, or responsible for 
delivering the information to the intended recipient, you are hereby notified 
that any dissemination, distribution, printing or copying of this e-mail and 
any attachments is strictly prohibited. If this e-mail and any attachments were 
received in error, please notify the sender by reply e-mail and delete the 
original message.

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Andrey V. Elsukov]

13.06.2022 21:25, Mike Jakubik пишет:

Hello,

I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, 
however, I am unable to get much more than 6Gb/s when testing with iperf3.


The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 
GB RAM, Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)


They are connected to a Dell N3224PX-ON switch. Both servers are idle 
and not in use, with a fresh install of stable/13-ebea872f8, nothing 
running on them except ssh, sendmail, etc.


The same exact servers tested on Linux (fedora 34) produced nearly 3x 
the performance (see attached screenshots), i was able to get a steady 
14.6Gb/s rate with nearly 0 retries shown in iperf, the performance on 
FreeBSD seems to avg at around 6Gbs but it is very sporadic during the 
iperf run.


# ifconfig mce0
mce0: flags=8863 metric 0 mtu 1500
options=ffed07bb
     ether b8:ce:f6:81:df:6a
     inet 192.168.10.31 netmask 0xff00 broadcast 192.168.10.255
     media: Ethernet 25GBase-CR 
     status: active
     nd6 options=29


Hi,

Do you have the same MTU size on linux machine?

--
WBR, Andrey V. Elsukov


OpenPGP_signature
Description: OpenPGP digital signature

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Hans Petter Selasky]

On 6/13/22 20:25, Mike Jakubik wrote:

Hello,

I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, 
however, I am unable to get much more than 6Gb/s when testing with iperf3.



The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, 
Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)



They are connected to a Dell N3224PX-ON switch. Both servers are idle and not 
in use, with a fresh install of stable/13-ebea872f8, nothing running on them 
except ssh, sendmail, etc.



When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I 
have tried just about every knob listed in 
https://calomel.org/freebsd_network_tuning.html with little impact on the 
performance. The network cards have HW LRO enabled as per the driver 
documentation (though this only seems to lower IRQ usage with no impact on 
actual throughput).



The same exact servers tested on Linux (fedora 34) produced nearly 3x the 
performance (see attached screenshots), i was able to get a steady 14.6Gb/s 
rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to 
avg at around 6Gbs but it is very sporadic during the iperf run.



I have run out of ideas, any suggestions are welcome. Considering Netflix uses 
very similar HW and they push 400 Gb/s tells me there is something really wrong 
here or Netflix isnt sharing all their secret sauce.




Some ideas:

Try to disable "rxpause,txpause" when setting the media.

Keep HW LRO off for now, it doesn't work for large number of connections.

What is the CPU usage during test? Is iperf3 running on a CPU-core which 
has direct access to the NIC's numa domain?


Is the NIC installed in the "correct" PCI high-performance slot?

There are some sysctl knobs which may tell where the problem is, if it's 
PCI backpressure or something else.


sysctl -a | grep diag_pci_enable
sysctl -a | grep diag_general_enable

Set these two to 1, then run some traffic and dump all mce sysctls:

sysctl -a | grep mce > dump.txt

--HPS

Re: Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Hi,



No, I do not see any retransmission in Linux (see the forum URL for 
screenshots) so I do not think this is a hardware issue. I don't think these 
cards have flow control on them. I also do not see any errors, drops, or 
collisions in netstat -i. It's like the network stack doesnt know what do do 
initially, it seems to sometimes even out after a few seconds, see below. In 
Linux I get instant 14.6Gb and it stays that way, with zero retries.



[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 42022 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec   623 MBytes  5.23 Gbits/sec  171    640 KBytes

[  5]   1.00-2.00   sec   613 MBytes  5.14 Gbits/sec  135    543 KBytes

[  5]   2.00-3.00   sec   662 MBytes  5.55 Gbits/sec  107    471 KBytes

[  5]   3.00-4.00   sec   718 MBytes  6.02 Gbits/sec   32    350 KBytes

[  5]   4.00-5.00   sec   709 MBytes  5.95 Gbits/sec   28    685 KBytes

[  5]   5.00-6.00   sec   713 MBytes  5.98 Gbits/sec   39    603 KBytes

[  5]   6.00-7.00   sec   704 MBytes  5.91 Gbits/sec   95    540 KBytes

[  5]   7.00-8.00   sec   716 MBytes  6.01 Gbits/sec   49    466 KBytes

[  5]   8.00-9.00   sec   722 MBytes  6.06 Gbits/sec  132    752 KBytes

[  5]   9.00-10.00  sec   720 MBytes  6.04 Gbits/sec   19    649 KBytes

[  5]  10.00-11.00  sec   720 MBytes  6.04 Gbits/sec  267    474 KBytes

[  5]  11.00-12.00  sec   675 MBytes  5.65 Gbits/sec  138   1.16 MBytes

[  5]  12.00-13.00  sec  1.04 GBytes  8.96 Gbits/sec  118   1.22 MBytes

[  5]  13.00-14.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.29 MBytes

[  5]  14.00-15.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.31 MBytes

[  5]  15.00-16.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.34 MBytes

[  5]  16.00-17.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.34 MBytes

[  5]  17.00-18.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.36 MBytes

[  5]  18.00-19.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.36 MBytes

[  5]  19.00-20.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.37 MBytes

[  5]  20.00-21.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.39 MBytes

[  5]  21.00-22.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.40 MBytes

[  5]  22.00-23.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.41 MBytes

[  5]  23.00-24.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.41 MBytes

[  5]  24.00-25.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.42 MBytes

[  5]  25.00-26.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.44 MBytes

[  5]  26.00-27.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.44 MBytes

[  5]  27.00-28.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.44 MBytes

[  5]  28.00-29.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.45 MBytes

[  5]  29.00-30.00  sec  1.29 GBytes  11.1 Gbits/sec    0   1.46 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  31.1 GBytes  8.91 Gbits/sec  1330 sender

[  5]   0.00-30.00  sec  31.1 GBytes  8.91 Gbits/sec  receiver





Thanks.







 On Mon, 13 Jun 2022 14:41:05 -0400 Santiago Martinez 
 wrote 








Mike Jakubik

https://www.swiftsmsgateway.com/



Disclaimer: This e-mail and any attachments are intended only for the use of 
the addressee(s) and may contain information that is privileged or 
confidential. If you are not the intended recipient, or responsible for 
delivering the information to the intended recipient, you are hereby notified 
that any dissemination, distribution, printing or copying of this e-mail and 
any attachments is strictly prohibited. If this e-mail and any attachments were 
received in error, please notify the sender by reply e-mail and delete the 
original message.








Hi there, there are a lot of re-transmission there... do you see
  the same with Linux? 

Are you seeing any drops or error counters increasing on the
  switch side? 

Have you check the sysctl for the card, I never used mellanox,
  but im pretty sure people here can help you.

You can also give it a try disabling control flow.

Hope it helps.

Santi



On 6/13/22 20:25, Mike Jakubik wrote:




Hello,



I have two new servers with a Mellnox ConnectX-6 card
  linked at 25Gb/s, however, I am unable to get much more
  than 6Gb/s when testing with iperf3.



The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core
  Processor, 256 GB RAM, Mellanox ConnectX-6 Lx 10/25GbE
  SFP28 2-port OCP Ethernet Adapter)



They are connected to a Dell N3224PX-ON switch. Both
  servers are idle and not in use, with a fresh install
  of stable/13-ebea872f8, nothing running on them except
  ssh, sendmail, etc.



When i test with iperf3 I am unable to get a higher avg
  than about 6Gb/s. I have tried just about every 

Poor performance with stable/13 and Mellanox ConnectX-6 (mlx5)

[Mike Jakubik]

Hello,



I have two new servers with a Mellnox ConnectX-6 card linked at 25Gb/s, 
however, I am unable to get much more than 6Gb/s when testing with iperf3.



The servers are Lenovo SR665 (2 x AMD EPYC 7443 24-Core Processor, 256 GB RAM, 
Mellanox ConnectX-6 Lx 10/25GbE SFP28 2-port OCP Ethernet Adapter)



They are connected to a Dell N3224PX-ON switch. Both servers are idle and not 
in use, with a fresh install of stable/13-ebea872f8, nothing running on them 
except ssh, sendmail, etc.



When i test with iperf3 I am unable to get a higher avg than about 6Gb/s. I 
have tried just about every knob listed in 
https://calomel.org/freebsd_network_tuning.html with little impact on the 
performance. The network cards have HW LRO enabled as per the driver 
documentation (though this only seems to lower IRQ usage with no impact on 
actual throughput).



The same exact servers tested on Linux (fedora 34) produced nearly 3x the 
performance (see attached screenshots), i was able to get a steady 14.6Gb/s 
rate with nearly 0 retries shown in iperf, the performance on FreeBSD seems to 
avg at around 6Gbs but it is very sporadic during the iperf run.



I have run out of ideas, any suggestions are welcome. Considering Netflix uses 
very similar HW and they push 400 Gb/s tells me there is something really wrong 
here or Netflix isnt sharing all their secret sauce.





# ifconfig mce0

mce0: flags=8863 metric 0 mtu 1500 

options=ffed07bb

    ether b8:ce:f6:81:df:6a

    inet 192.168.10.31 netmask 0xff00 broadcast 192.168.10.255

    media: Ethernet 25GBase-CR 

    status: active

    nd6 options=29





[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01 

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 64695 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec   930 MBytes  7.80 Gbits/sec   62    789 KBytes

[  5]   1.00-2.00   sec   942 MBytes  7.90 Gbits/sec  164    824 KBytes

[  5]   2.00-3.00   sec  1.00 GBytes  8.61 Gbits/sec  402    879 KBytes

[  5]   3.00-4.00   sec   761 MBytes  6.39 Gbits/sec   61    588 KBytes

[  5]   4.00-5.00   sec   724 MBytes  6.07 Gbits/sec  220    497 KBytes

[  5]   5.00-6.00   sec   723 MBytes  6.07 Gbits/sec   54    364 KBytes

[  5]   6.00-7.00   sec   716 MBytes  6.01 Gbits/sec  187    682 KBytes

[  5]   7.00-8.00   sec   728 MBytes  6.11 Gbits/sec   86    568 KBytes

[  5]   8.00-9.00   sec   761 MBytes  6.39 Gbits/sec   37    418 KBytes

[  5]   9.00-10.00  sec   733 MBytes  6.15 Gbits/sec    8    617 KBytes

[  5]  10.00-11.00  sec   734 MBytes  6.16 Gbits/sec  238    474 KBytes

[  5]  11.00-12.00  sec   736 MBytes  6.17 Gbits/sec  164    757 KBytes

[  5]  12.00-13.00  sec   610 MBytes  5.12 Gbits/sec  118    579 KBytes

[  5]  13.00-14.00  sec  1.02 GBytes  8.75 Gbits/sec  447    449 KBytes

[  5]  14.00-15.00  sec   728 MBytes  6.11 Gbits/sec  132    719 KBytes

[  5]  15.00-16.00  sec   724 MBytes  6.07 Gbits/sec  185    649 KBytes

[  5]  16.00-17.00  sec   597 MBytes  5.01 Gbits/sec  142    570 KBytes

[  5]  17.00-18.00  sec   733 MBytes  6.15 Gbits/sec  102    484 KBytes

[  5]  18.00-19.00  sec   726 MBytes  6.09 Gbits/sec   15    569 KBytes

[  5]  19.00-20.00  sec   733 MBytes  6.15 Gbits/sec  181    527 KBytes

[  5]  20.00-21.00  sec   729 MBytes  6.12 Gbits/sec  118    430 KBytes

[  5]  21.00-22.00  sec   733 MBytes  6.15 Gbits/sec  116    641 KBytes

[  5]  22.00-23.00  sec   728 MBytes  6.10 Gbits/sec  182    598 KBytes

[  5]  23.00-24.00  sec   743 MBytes  6.24 Gbits/sec  209    614 KBytes

[  5]  24.00-25.00  sec   746 MBytes  6.26 Gbits/sec   72    758 KBytes

[  5]  25.00-26.00  sec   742 MBytes  6.23 Gbits/sec  199    675 KBytes

[  5]  26.00-27.00  sec   799 MBytes  6.70 Gbits/sec  183    542 KBytes

[  5]  27.00-28.00  sec   908 MBytes  7.61 Gbits/sec    7   1.19 MBytes

[  5]  28.00-29.00  sec  1.37 GBytes  11.7 Gbits/sec  606   1013 KBytes

[  5]  29.00-30.00  sec  1.31 GBytes  11.3 Gbits/sec   74   1.02 MBytes

- - - - - - - - - - - - - - - - - - - - - - - - -

[ ID] Interval   Transfer Bitrate Retr

[  5]   0.00-30.00  sec  23.7 GBytes  6.79 Gbits/sec  4771 sender

[  5]   0.00-30.00  sec  23.7 GBytes  6.79 Gbits/sec  receiver





I have even tried changing to the RACK TCP stack, only to get slightly better 
results, however with RACK the amount of retries is nearly 0.



[root@db-02 ~]# sysctl net.inet.tcp.functions_default=rack 

net.inet.tcp.functions_default: rack -> rack

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01

[root@db-02 ~]# iperf3 -i 1 -t 30 -c db-01

Connecting to host db-01, port 5201

[  5] local 192.168.10.31 port 51894 connected to 192.168.10.30 port 5201

[ ID] Interval   Transfer Bitrate Retr  Cwnd

[  5]   0.00-1.00   sec   761 MBytes  6.38 Gbits/sec    0    737 KBytes

[  5]   1.00-2.00   sec   859 MBytes  7.21 Gbits/sec    0    761 KBytes