Hi tech@.
Two machines (A and B) running recent 5.4-stable plugged into same switch.
A has:
em0 at pci4 dev 0 function 0 "Intel 82573E" rev 0x03: msi, address
00:30:48:66:a0:ec
em1 at pci5 dev 0 function 0 "Intel 82573L" rev 0x00: msi, address
00:30:48:66:a0:ed
B has:
bnx0 at pci2 dev 0 function 0 "Broadcom BCM5716" rev 0x20: apic 0 int 16
bnx1 at pci2 dev 0 function 1 "Broadcom BCM5716" rev 0x20: apic 0 int 17
bnx0: address b8:ac:6f:91:48:da
brgphy0 at bnx0 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
bnx1: address b8:ac:6f:91:48:db
brgphy1 at bnx1 phy 1: BCM5709 10/100/1000baseT PHY, rev. 8
Both servers have LACP trunk(4)s built on-top the above mentioned interfaces:
A has:
em0:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6>
mtu 1500
lladdr 00:30:48:66:a0:ec
priority: 0
trunk: trunkdev trunk0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
em1:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6>
mtu 1500
lladdr 00:30:48:66:a0:ec
priority: 0
trunk: trunkdev trunk0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
trunk0: flags=28943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,NOINET6>
mtu 1500
lladdr 00:30:48:66:a0:ec
priority: 0
trunk: trunkproto lacp
trunk id: [(8000,00:30:48:66:a0:ec,402C,0000,0000),
(0001,ec:30:91:25:c0:4f,03E8,0000,0000)]
trunkport em1 active,collecting,distributing
trunkport em0 active,collecting,distributing
groups: trunk
media: Ethernet autoselect
status: active
B has:
bnx0:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6>
mtu 1500
lladdr b8:ac:6f:91:48:da
priority: 0
trunk: trunkdev trunk0
media: Ethernet autoselect (1000baseT full-duplex)
status: active
bnx1:
flags=28b43<UP,BROADCAST,RUNNING,PROMISC,ALLMULTI,SIMPLEX,MULTICAST,NOINET6>
mtu 1500
lladdr b8:ac:6f:91:48:da
priority: 0
trunk: trunkdev trunk0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
trunk0: flags=28843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,NOINET6> mtu 1500
lladdr b8:ac:6f:91:48:da
priority: 0
trunk: trunkproto lacp
trunk id: [(8000,b8:ac:6f:91:48:da,402C,0000,0000),
(0001,ec:30:91:25:c0:4f,03EA,0000,0000)]
trunkport bnx1 active,collecting,distributing
trunkport bnx0 active,collecting,distributing
groups: trunk
media: Ethernet autoselect
status: active
Now about the difference.
The A receives on both em0 and em1 and transmits on em0:
IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS
em0 up:U 2711 2859K 0 5593 5222K 0 0
em1 up:U 2867 2343K 0 10 3226 0 0
trunk0 up:U 5578 5202K 0 5603 5225K 0 0
The B receives *only* on bnx0 and transmits *only* on bnx1:
IFACE STATE DESC IPKTS IBYTES IERRS OPKTS OBYTES OERRS COLLS
bnx0 up:U 2873 2956K 0 2 977 0 0
bnx1 up:U 5 360 0 3119 2604K 0 0
trunk0 up:U 2878 2956K 0 3121 2605K 0 0
The only difference ifconfig shows, both em(4)s are master interfaces
on A, but only
bnx1 is a master interface on B (I haven't found any description of
"master" media option
in ifconfig man page, trunk man page saying about "master" but only
wrt failover mode).
Whole situation smells like trunk(4) receives on *all* master
interfaces, but transmits
on *first available* master.
The question here is, why both em(4)s are master interfaces on A, but
only bnx1 is
master interface on B?
Another question is, what is transmit hash policy for a trunk in LACP
mode? If it matters,
while testing on B, different MACs and different VLANs been used, but
effect is same:
bnx0 only receives, bnx1 only transmits.
Anybody with trunking experience please speak up.
Thanks,
Alexey
