Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On Mon, Nov 11, 2013 at 11:42 AM, Stuart Henderson s...@spacehopper.org wrote: master on em0/em1/bnx0 is nothing to do with trunk, it is about the gigabit ethernet clocking source. ok, but it is obvious: documentation is unclear (silent) about that. lacp hashing policy is the same as for loadbalance, see the manpage and confirm in trunk_hashmbuf(). I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem.
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
I'm not sure if I am misunderstanding your direction of inbound, but that would be an effect of what the switch does, would it not? If the switch isn't configured for LACP correctly, then it would send the traffic to one of them, only. 2013/11/11 Alexey Suslikov alexey.susli...@gmail.com On Mon, Nov 11, 2013 at 11:42 AM, Stuart Henderson s...@spacehopper.org wrote: master on em0/em1/bnx0 is nothing to do with trunk, it is about the gigabit ethernet clocking source. ok, but it is obvious: documentation is unclear (silent) about that. lacp hashing policy is the same as for loadbalance, see the manpage and confirm in trunk_hashmbuf(). I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. -- May the most significant bit of your life be positive.
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On Mon, Nov 11, 2013 at 12:19 PM, Janne Johansson icepic...@gmail.com wrote: I'm not sure if I am misunderstanding your direction of inbound, but that would be an effect of what the switch does, would it not? If the switch isn't configured for LACP correctly, then it would send the traffic to one of them, only. again, consider the following output IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS bnx0 up:U2873 2956K 0 2977 0 0 bnx1 up:U 5360 0 3119 2604K 0 0 trunk0up:U2878 2956K 0 3121 2605K 0 0 (inbound is distributed via single interface, outbound - via 2nd interface in trunk) IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS em0 up:U2711 2859K 0 5593 5222K 0 0 em1 up:U2867 2343K 0 10 3226 0 0 trunk0up:U5578 5202K 0 5603 5225K 0 0 (inbound is distributed via both interfaces, outbound - via 1st interface in trunk) I'm less worried about outbound, however it is interesting why em(4) setup uses first interface, but bnx(4) setup uses second. by 1st and 2nd I mean an order of addition inside hostname.if $ cat /etc/hostname.trunk0 trunkproto lacp trunkport bnx0 trunkport bnx1 up -inet6 $ cat /etc/hostname.trunk0 trunkproto lacp trunkport em0 trunkport em1 up -inet6 on switch itself, both trunks have no visible difference in configuration. 2013/11/11 Alexey Suslikov alexey.susli...@gmail.com On Mon, Nov 11, 2013 at 11:42 AM, Stuart Henderson s...@spacehopper.org wrote: master on em0/em1/bnx0 is nothing to do with trunk, it is about the gigabit ethernet clocking source. ok, but it is obvious: documentation is unclear (silent) about that. lacp hashing policy is the same as for loadbalance, see the manpage and confirm in trunk_hashmbuf(). I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. -- May the most significant bit of your life be positive.
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On 2013/11/11 12:15, Alexey Suslikov wrote: On Mon, Nov 11, 2013 at 11:42 AM, Stuart Henderson s...@spacehopper.org wrote: master on em0/em1/bnx0 is nothing to do with trunk, it is about the gigabit ethernet clocking source. ok, but it is obvious: documentation is unclear (silent) about that. Why would something listed as a media characteristic of the physical interface have anything to do with trunk? lacp hashing policy is the same as for loadbalance, see the manpage and confirm in trunk_hashmbuf(). I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. The trunk driver can't influence inbound packet distribution, that is down to the device sending packets e.g. your switch..
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On Mon, Nov 11, 2013 at 12:43 PM, Stuart Henderson st...@openbsd.org wrote: On 2013/11/11 12:15, Alexey Suslikov wrote: On Mon, Nov 11, 2013 at 11:42 AM, Stuart Henderson s...@spacehopper.org wrote: master on em0/em1/bnx0 is nothing to do with trunk, it is about the gigabit ethernet clocking source. ok, but it is obvious: documentation is unclear (silent) about that. Why would something listed as a media characteristic of the physical interface have anything to do with trunk? well, I just expected to see master media option documented somewhere, to make it clear what is trunk master and what is clocking master. lacp hashing policy is the same as for loadbalance, see the manpage and confirm in trunk_hashmbuf(). I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. The trunk driver can't influence inbound packet distribution, that is down to the device sending packets e.g. your switch.. yes, I know. but bnx(4) interfaces have master set differently, in contrast to em(4) interfaces. I'm really guessing, but maybe that clocking source has some effect for a switch.
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On 2013/11/11 12:15, Alexey Suslikov wrote: I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. On 2013/11/11 10:43, I wrote: The trunk driver can't influence inbound packet distribution, that is down to the device sending packets e.g. your switch.. ... for newer HP L3 switches, you might want to look at trunk-load-balance L4-based, for ciscos port-channel load-balance..
Re: 2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
On Mon, Nov 11, 2013 at 1:00 PM, Stuart Henderson st...@openbsd.org wrote: On 2013/11/11 12:15, Alexey Suslikov wrote: I see different inbound packet distribution on trunk on-top of em(4)s and on trunk on top of bnx(4)s - that's the real problem. On 2013/11/11 10:43, I wrote: The trunk driver can't influence inbound packet distribution, that is down to the device sending packets e.g. your switch.. ... for newer HP L3 switches, you might want to look at trunk-load-balance L4-based, for ciscos port-channel load-balance.. yes, I'm aware of above options, but I have SPS2024-G5 in this setup. did some tests with mode servers involved, and outbound is no worry. this is how trunk(4) hashing works. IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS bnx0 up:U 487 237275 0129 41107 0 0 bnx1 up:U 5360 0348 65383 0 0 IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS em0 up:U 228 54112 0136 51470 0 0 em1 up:U 218 65348 0322 79837 0 0 but bnx1 inbound is always showing 4-5 packets no matter how traffic is distributed :/
2 x em(4) and 2 x bnx(4) trunk(4)s behave differently
Hi tech@. Two machines (A and B) running recent 5.4-stable plugged into same switch. A has: em0 at pci4 dev 0 function 0 Intel 82573E rev 0x03: msi, address 00:30:48:66:a0:ec em1 at pci5 dev 0 function 0 Intel 82573L rev 0x00: msi, address 00:30:48:66:a0:ed B has: bnx0 at pci2 dev 0 function 0 Broadcom BCM5716 rev 0x20: apic 0 int 16 bnx1 at pci2 dev 0 function 1 Broadcom BCM5716 rev 0x20: apic 0 int 17 bnx0: address b8:ac:6f:91:48:da brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 bnx1: address b8:ac:6f:91:48:db brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8 Both servers have LACP trunk(4)s built on-top the above mentioned interfaces: A has: em0: flags=28b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active em1: flags=28b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active trunk0: flags=28943UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr 00:30:48:66:a0:ec priority: 0 trunk: trunkproto lacp trunk id: [(8000,00:30:48:66:a0:ec,402C,,), (0001,ec:30:91:25:c0:4f,03E8,,)] trunkport em1 active,collecting,distributing trunkport em0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active B has: bnx0: flags=28b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex) status: active bnx1: flags=28b43UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkdev trunk0 media: Ethernet autoselect (1000baseT full-duplex,master) status: active trunk0: flags=28843UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6 mtu 1500 lladdr b8:ac:6f:91:48:da priority: 0 trunk: trunkproto lacp trunk id: [(8000,b8:ac:6f:91:48:da,402C,,), (0001,ec:30:91:25:c0:4f,03EA,,)] trunkport bnx1 active,collecting,distributing trunkport bnx0 active,collecting,distributing groups: trunk media: Ethernet autoselect status: active Now about the difference. The A receives on both em0 and em1 and transmits on em0: IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS em0 up:U2711 2859K 0 5593 5222K 0 0 em1 up:U2867 2343K 0 10 3226 0 0 trunk0up:U5578 5202K 0 5603 5225K 0 0 The B receives *only* on bnx0 and transmits *only* on bnx1: IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS bnx0 up:U2873 2956K 0 2977 0 0 bnx1 up:U 5360 0 3119 2604K 0 0 trunk0up:U2878 2956K 0 3121 2605K 0 0 The only difference ifconfig shows, both em(4)s are master interfaces on A, but only bnx1 is a master interface on B (I haven't found any description of master media option in ifconfig man page, trunk man page saying about master but only wrt failover mode). Whole situation smells like trunk(4) receives on *all* master interfaces, but transmits on *first available* master. The question here is, why both em(4)s are master interfaces on A, but only bnx1 is master interface on B? Another question is, what is transmit hash policy for a trunk in LACP mode? If it matters, while testing on B, different MACs and different VLANs been used, but effect is same: bnx0 only receives, bnx1 only transmits. Anybody with trunking experience please speak up. Thanks, Alexey
