https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=234242
Bug ID: 234242
Summary: LACP l2,l3,l4 load sharing only respects dst port
Product: Base System
Version: 11.2-STABLE
Hardware: amd64
OS: Any
Status: New
Severity: Affects Many People
Priority: ---
Component: kern
Assignee: [email protected]
Reporter: [email protected]
Hi,
I'm chasing a performance problem when using FreeBSD as a router.
The system has 4 Mellanox ConnectX-3 cards. Bonded to 2 LAGGs with LACP.
mlxen0 and 1 are lagg0 and mlxen2 and 3 are lagg1.
Directly on these interface are linux boxes also with ConnectX-3 cards.
This is the ifconfig from the router:
mlxen0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5b:30
hwaddr 24:8a:07:f7:5b:30
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
mlxen1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5b:30
hwaddr 24:8a:07:f7:5b:31
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5f:10
hwaddr 24:8a:07:f7:5f:10
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
mlxen3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5f:10
hwaddr 24:8a:07:f7:5f:11
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect (10Gbase-CX4 <full-duplex,rxpause,txpause>)
status: active
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5b:30
inet6 fe80::268a:7ff:fef7:5b30%lagg0 prefixlen 64 scopeid 0xb
inet 10.22.1.1 netmask 0xffffff00 broadcast 10.22.1.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: mlxen0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: mlxen1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
lagg1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8d00b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE>
ether 24:8a:07:f7:5f:10
inet6 fe80::268a:7ff:fef7:5f10%lagg1 prefixlen 64 scopeid 0xc
inet 10.22.2.1 netmask 0xffffff00 broadcast 10.22.2.255
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
media: Ethernet autoselect
status: active
groups: lagg
laggproto lacp lagghash l2,l3,l4
laggport: mlxen2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
laggport: mlxen3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
When I put the 2 linux boxes directly together I can easily achive 20GBits.
iperf server: iperf3 -V -p 5000 -f m -s
iperf client: iperf3 -p 5000 -f m -V -c 10.22.2.10 -t 30 -P 10
When I put the BSD router between I can only achieve 10Gbit.
>From the man page of ifconfig there states:
lagghash option[,option]
Set the packet layers to hash for aggregation protocols which
load balance. The default is "l2,l3,l4". The options can be
combined using commas.
l2 src/dst mac address and optional vlan number.
l3 src/dst address for IPv4 or IPv6.
l4 src/dst port for TCP/UDP/SCTP.
The problem is that l4 is not really true, because iperf on multistream (-P 10)
uses multiple source ports, but the load comes from lagg0 with 5Gbit divided on
mlxen0 and mlxen1 and goes via lagg1 through one of the interfaces with 10Gbit.
If I start a second iperf instance, listening on a different port and start
both, the traffic flows with 20Gbit correctly shared.
Searching bugtracker, forums and asking IRC doesn't gave any good answer,
already played with lacp strict mode and enable/disable flowid does not help.
Right now I'm not sure if this is a bug in kernel or documentation, but it
would be cool if we can include src and dst ports in hashing calculation.
Thanks
Michael
--
You are receiving this mail because:
You are the assignee for the bug.
_______________________________________________
[email protected] mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-bugs
To unsubscribe, send any mail to "[email protected]"
