On 10.6.2022. 10:20, David Gwynne wrote:
>
>
>> On 10 Jun 2022, at 08:24, Hrvoje Popovski <hrv...@srce.hr> wrote:
>>
>> On 9.6.2022. 19:25, Hrvoje Popovski wrote:
>>> On 9.6.2022. 19:11, Jan Klemkow wrote:
>>>> On Thu, Jun 09, 2022 at 08:25:22AM +0200, Hrvoje Popovski wrote:
>>>>> On 8.6.2022. 22:01, Hrvoje Popovski wrote:
>>>>>> On 8.6.2022. 15:04, Jan Klemkow wrote:
>>>>>>> Could you show me, how your setup and your configuration looks like?
>>>>>> Yes, of course ..
>>>>>>
>>>>>> All my lab boxes are connected to switch (no flow-control). In this
>>>>>> setup ix0 and ix1 are in aggr and vlans 700 and 800 are tagged on aggr3.
>>>>>> I have tried tcpbench and iperf3 from openbsd or linux box and with tso
>>>>>> I'm getting few Kbps. In attachment you can find tcpdump for iperf3 on
>>>>>> openbsd with or without tso
>>>>>>
>>>>>> I will play more with this setup maybe I'm doing something terribly
>>>>>> wrong...
>>>>>>
>>>>>> My OpenBSD conf:
>>>>>>
>>>>>> r620-1# cat /etc/hostname.ix0
>>>>>> up
>>>>>>
>>>>>> r620-1# cat /etc/hostname.ix1
>>>>>> up
>>>>>>
>>>>>> r620-1# cat /etc/hostname.aggr3
>>>>>> trunkport ix0
>>>>>> trunkport ix1
>>>>>> up
>>>>>>
>>>>>> r620-1# cat /etc/hostname.vlan700
>>>>>> parent aggr3
>>>>>> vnetid 700
>>>>>> lladdr ec:f4:bb:da:f7:f8
>>>>>> inet 192.168.100.11/24
>>>>>> !route add 16/8 192.168.100.111
>>>>>> up
>>>>>>
>>>>>> r620-1# cat /etc/hostname.vlan800
>>>>>> parent aggr3
>>>>>> vnetid 800
>>>>>> lladdr ec:f4:bb:da:f7:fa
>>>>>> inet 192.168.111.11/24
>>>>>> !route add 48/8 192.168.111.111
>>>>>> up
>>>>>
>>>>> now I've configured aggr like this
>>>>>
>>>>> ix0 - aggr0 - vlan700
>>>>> ix1 - aggr1 - vlan800
>>>>> ip configuration is the same
>>>>>
>>>>> still I have problem with tso
>>>>>
>>>>> tcpdump in attachment
>>>>
>>>> The tcpdump output on my machine looks similar to yours.
>>>> Do you see TCP retransmits on your sending Linux host?
>>>>
>>>> # netstat -s | grep retransmitted
>>>> 0 segments retransmitted
>>>>
>>>
>>> [root@test2 pktgen]# netstat -s | grep -i segment
>>> 8352173627 segments received
>>> 14648977712 segments send out
>>> 13336873 segments retransmited
>>> 586 bad segments received.
>>>
>>> yes, segments retransmited is increasing
>>>
>>>
>>>> Which specific ix(4) NIC do you use?
>>>>
>>>
>>> r620-1# dmesg | grep ix
>>> mtrr: Pentium Pro MTRR support, 10 var ranges, 88 fixed ranges
>>> ix0 at pci1 dev 0 function 0 "Intel 82599" rev 0x01, msix, 6 queues,
>>> address ec:f4:bb:da:f7:f8
>>> ix1 at pci1 dev 0 function 1 "Intel 82599" rev 0x01, msix, 6 queues,
>>> address ec:f4:bb:da:f7:fa
>>> ix2 at pci4 dev 0 function 0 "Intel X540T" rev 0x01, msix, 6 queues,
>>> address a0:36:9f:29:f2:0c
>>> ix3 at pci4 dev 0 function 1 "Intel X540T" rev 0x01, msix, 6 queues,
>>> address a0:36:9f:29:f2:0e
>>>
>>>
>>> I'm using ix0 and ix1 for testing
>>>> I'm using these two:
>>>>
>>>> # dmesg | grep ^ix
>>>> ix0 at pci3 dev 0 function 0 "Intel 82599" rev 0x01, msix, 12 queues,
>>>> address 00:1b:21:87:fb:2c
>>>> ix1 at pci3 dev 0 function 1 "Intel 82599" rev 0x01, msix, 12 queues,
>>>> address 00:1b:21:87:fb:2d
>>>>
>>>> Thanks,
>>>> Jan
>>>>
>>>
>>
>> Hi,
>>
>> I tried with trunk lacp and tagged vlan700 over trunk0 and it's working
>> as expected. Everything is the same as before, only aggr0 is now trunk0.
>>
>> with tso
>> [ 4] 17.00-18.00 sec 1.09 GBytes 9.39 Gbits/sec 22 622 KBytes
>> [ 4] 18.00-19.00 sec 1.09 GBytes 9.39 Gbits/sec 17 662 KBytes
>> [ 4] 19.00-20.00 sec 1.09 GBytes 9.39 Gbits/sec 0 696 KBytes
>> [ 4] 20.00-21.00 sec 1.09 GBytes 9.39 Gbits/sec 0 735 KBytes
>>
>>
>> without tso
>> [ 4] 17.00-18.00 sec 594 MBytes 4.98 Gbits/sec 0 612 KBytes
>> [ 4] 18.00-19.00 sec 594 MBytes 4.98 Gbits/sec 0 646 KBytes
>> [ 4] 19.00-20.00 sec 595 MBytes 4.99 Gbits/sec 0 665 KBytes
>> [ 4] 20.00-20.87 sec 515 MBytes 4.99 Gbits/sec 0 675 KBytes
>>
>>
>> tcpdump is the same as with aggr...
>>
>>
>> After this I've tried same aggr/vlan setup on two different boxes with
>> ix interfaces and send traffic to them from linux and openbsd and
>> results are not good ...
>> Only differences between your setup and mine is that i have switch and
>> you have directly connected hosts... but i don't think that should be
>> the problem..
>>
>> It would be great if someone else could test this because it's not that
>> i don't know what I'm doing but I'm running out of ideas...
>
> is this rx ring moderation not scaling fast enough?
>
> what's systat mb look like when you're trying? do you see qdrops increase in
> the ix-rxq kstats?
>
> dlg
while sending traffic from linux to openbsd ix0 with tso
systat mb
IFACE RING LIVELOCKS SIZE ALIVE LWM HWM CWM
System mbufs 0 256 1685 143
mcl2k 2048 513 74
mcl2k2 2112 615 82
mcl4k 4096 0 1
mcl8k 8192 512 65
mcl12k 12288 36 10
ix0 0 2048 12 10 255 12
1 2048 13 10 255 13
2 2048 10 10 255 10
3 2048 10 10 255 10
4 2048 10 10 255 10
5 2048 10 10 255 10
6 2048 10 10 255 10
7 2048 10 10 255 10
8 2048 13 10 255 13
9 2048 10 10 255 10
10 2048 11 10 255 11
11 2048 10 10 255 10
12 2048 13 10 255 13
13 2048 10 10 255 10
14 2048 12 10 255 12
15 2048 10 10 255 10
I don't see any qdrops increasing
smc24# kstat ix-rxq | grep qdrops
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
qdrops: 0 packets
full kstat for ix0 while sending traffic
ix0:0:ix-rxq:0
packets: 2637 packets
bytes: 347878 bytes
qdrops: 0 packets
ix0:0:ix-rxq:1
packets: 237 packets
bytes: 355434 bytes
qdrops: 0 packets
ix0:0:ix-rxq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:4
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:5
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:6
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:7
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:8
packets: 21023 packets
bytes: 31908615 bytes
qdrops: 0 packets
ix0:0:ix-rxq:9
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:10
packets: 8 packets
bytes: 693 bytes
qdrops: 0 packets
ix0:0:ix-rxq:11
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
ix0:0:ix-rxq:12
packets: 599 packets
bytes: 903535 bytes
qdrops: 0 packets
ix0:0:ix-rxq:13
packets: 2494545 packets
bytes: 3771018781 bytes
qdrops: 0 packets
ix0:0:ix-rxq:14
packets: 14417821 packets
bytes: 21806454328 bytes
qdrops: 0 packets
ix0:0:ix-rxq:15
packets: 12 packets
bytes: 993 bytes
qdrops: 0 packets
ix0:0:ix-stats:0
crc errs: 0 packets
tx link xon: 0 packets
rx link xon: 0 packets
tx link xoff: 0 packets
rx link xoff: 0 packets
rx 64B: 20 packets
rx 65-127B: 21423 packets
rx 128-255B: 5462 packets
rx 256-511B: 814 packets
rx 512-1023B: 2031 packets
rx 1024-maxB: 16907132 packets
tx good: 1911084 packets
rx good: 25678737785 bytes
tx good: 139295575 bytes
rx undersize: 0 packets
rx fragment: 0 packets
rx oversize: 0 packets
rx jabber: 0 packets
rx total: 25682300131 bytes
rx total: 16972733 packets
tx total: 1911084 packets
rx good: 16936882 packets
rx bcast: 37 packets
rx mcast: 2599 packets
tx 64B: 125 packets
tx 65-127B: 1908357 packets
tx 128-255B: 2602 packets
tx 256-511B: 0 packets
tx 512-1023B: 0 packets
tx 1024-maxB: 0 packets
tx mcast: 2603 packets
tx bcast: 3 packets
ix0:0:ix-txq:0
packets: 2738 packets
bytes: 396853 bytes
ix0:0:ix-txq:1
packets: 78 packets
bytes: 5580 bytes
ix0:0:ix-txq:2
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:3
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:4
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:5
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:6
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:7
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:8
packets: 13683 packets
bytes: 1055682 bytes
ix0:0:ix-txq:9
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:10
packets: 5 packets
bytes: 354 bytes
ix0:0:ix-txq:11
packets: 0 packets
bytes: 0 bytes
ix0:0:ix-txq:12
packets: 366 packets
bytes: 28272 bytes
ix0:0:ix-txq:13
packets: 1286487 packets
bytes: 90054066 bytes
ix0:0:ix-txq:14
packets: 607719 packets
bytes: 40109491 bytes
ix0:0:ix-txq:15
packets: 8 packets
bytes: 565 bytes
ix0:0:rxq:0
packets: 2637 packets
bytes: 347746 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:1
packets: 154 packets
bytes: 349008 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:4
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:5
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:6
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:7
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:8
packets: 16490 packets
bytes: 31525345 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:9
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:10
packets: 8 packets
bytes: 661 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:11
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:12
packets: 451 packets
bytes: 891371 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:13
packets: 2494545 packets
bytes: 3761040645 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:14
packets: 800101 packets
bytes: 20907684764 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:rxq:15
packets: 12 packets
bytes: 953 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
ix0:0:txq:0
packets: 2738 packets
bytes: 396397 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:1
packets: 78 packets
bytes: 5268 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:2
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:3
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:4
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:5
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:6
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:7
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:8
packets: 13683 packets
bytes: 1000950 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:9
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:10
packets: 5 packets
bytes: 334 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:11
packets: 0 packets
bytes: 0 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:12
packets: 366 packets
bytes: 26808 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:13
packets: 1286487 packets
bytes: 84908146 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:14
packets: 607719 packets
bytes: 40109459 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
ix0:0:txq:15
packets: 8 packets
bytes: 533 bytes
qdrops: 0 packets
errors: 0 packets
qlen: 0 packets
maxqlen: 255 packets
oactive: false
