On Wed, Aug 12, 2020 at 02:54:36PM +0100, Tom Smyth wrote:
> What is the Switch telling you about the LACP ?
This is a Cisco switch running Nexus not IOS so I am not quite as
familiar with it as the IOS ones so I may have a few things wrong here.
Also, you are asking for a lot of into so this is a long email.
This is what I think you are asking for.
> can you do a
>
> show port-channel
# sh port-channel
^
% Incomplete command at '^' marker
This is not right for NXOS. Do you mean this:
# show port-channel database
...
port-channel32
Last membership update is successful
2 ports in total, 2 ports up
First operational port is Ethernet5/38
Age of the port-channel is 5d:01h:54m:27s
Time since last bundle is 5d:01h:50m:54s
Last bundled member is Ethernet6/38
Ports: Ethernet5/38 [on] [up] *
Ethernet6/38 [on] [up]
port-channel33
Last membership update is successful
2 ports in total, 2 ports up
First operational port is Ethernet7/38
Age of the port-channel is 5d:01h:53m:58s
Time since last bundle is 5d:01h:48m:28s
Last bundled member is Ethernet8/38
Ports: Ethernet7/38 [on] [up] *
Ethernet8/38 [on] [up]
> show port-channel detailed
# sh port-channel detailed
^
% Invalid command at '^' marker.
Do you mean this:
show port-channel internal event-history all | include port-channel32 next 20
port-channel32
channel : 32
bundle : 65535
ifindex : 0x1600001f
admin mode : on
oper mode : on
nports : 2
active : 2
pre cfg : 0
ltl : 0x15 (21)
lif : 0x12bc
iod : 0x2bf (703)
global id : 20
flag : 0
lock count : 0
num. of SIs: 0
ac mbrs : 0 0
lacp graceful conv disable : 0
lacp suspend indiv disable : 0
pc min-links : 1
pc max-bundle : 32
--
PC SDB: Is a member of port-channel32
>>>>FSM: <Ethernet5/38> has 191 logged transitions<<<<<
>>>>
1) Event:ESQ_RSP length:38, at 75399 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153),
Opc:MTS_OPC_PIXM_MOD_MEMB_LTL(69637)
RRtoken:0x4C0F7380
2) Event:ESQ_REQ length:38, at 75466 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497)
RRtoken:0x4C0F73D9
3) Event:ESQ_RSP length:38, at 75593 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497)
RRtoken:0x4C0F73D9
4) Event:ESQ_REQ length:38, at 75646 usecs after Wed Aug 12 17:18:28 2020
--
PC SDB: Is a member of port-channel32
>>>>FSM: <Ethernet6/38> has 191 logged transitions<<<<<
>>>>
1) Event:ESQ_RSP length:38, at 263607 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153),
Opc:MTS_OPC_PIXM_MOD_MEMB_LTL(69637)
RRtoken:0x4C0F7603
2) Event:ESQ_REQ length:38, at 263663 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_TX] Dst:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497)
RRtoken:0x4C0F7613
3) Event:ESQ_RSP length:38, at 263825 usecs after Wed Aug 12 17:18:28 2020
Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS
[E_MTS_RX] Src:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497)
RRtoken:0x4C0F7613
4) Event:ESQ_REQ length:38, at 263887 usecs after Wed Aug 12 17:18:28 2020
Or maybe this:
# show port-channel internal info interface po32
port-channel32
channel : 32
bundle : 65535
ifindex : 0x1600001f
admin mode : on
oper mode : on
nports : 2
active : 2
pre cfg : 0
ltl : 0x15 (21)
lif : 0x12bc
iod : 0x2bf (703)
global id : 20
flag : 0
lock count : 0
num. of SIs: 0
ac mbrs : 0 0
lacp graceful conv disable : 0
lacp suspend indiv disable : 0
pc min-links : 1
pc max-bundle : 32
pc max active members : 32
pc is-suspend-minlinks : 0
port load defer enable : 0
port-channel bfd config enabled : 0
port-channel bfd config complete: 0
port-channel bfd destination: null
port-channel bfd start timeout: 0
lacp fast-select-hot-standby disable : 0
port-channel port hash-distribution : none
ethpm bundle lock count : 0
lacp delayed-enable fop Ethernet5/38 0x1a204a00
lacp delayed-enable : 0
lacp delayed-enable cfg-port none
lacp delayed-enable oper-port none
lacp delayed-enable local best priority : 0xffffffff
lacp delayed-enable remote best priority : 0xffffffff
lacp vpc conv enabled : 0
gir conv enabled : 0
bundle number map:
1-2Members:
Ethernet5/38 [bundle_no = 0] is_ltl_programmed = 1
Port BFD session state: 5 (none)
Ethernet6/38 [bundle_no = 1] is_ltl_programmed = 1
Port BFD session state: 5 (none)
port-channel external lock:
Lock Info: resource [eth-port-channel 32]
type[0] p_gwrap[(nil)]
FREE @ 657079 usecs after Fri Aug 7 15:33:46 2020
type[1] p_gwrap[(nil)]
FREE @ 967427 usecs after Wed Aug 12 17:08:17 2020
type[2] p_gwrap[(nil)]
FREE @ 393129 usecs after Fri Aug 7 15:37:19 2020
0x1600001f
internal (ethpm bundle) lock:
Lock Info: resource [eth-port-channel 32]
type[0] p_gwrap[(nil)]
FREE @ 657041 usecs after Fri Aug 7 15:33:46 2020
type[1] p_gwrap[(nil)]
FREE @ 456383 usecs after Wed Aug 12 17:26:29 2020
type[2] p_gwrap[(nil)]
FREE @ 455329 usecs after Wed Aug 12 17:26:29 2020
0x1600001f
> sho port-channel summary
sh port-channel summary
...
21 Po21(SU) Eth LACP Eth5/41(P) Eth6/41(P)
...
32 Po32(SU) Eth NONE Eth5/38(P) Eth6/38(P)
33 Po33(SU) Eth NONE Eth7/38(P) Eth8/38(P)
I included po21 here because that is OpenBSD 6.4 using trunk(4)
po33 is the box in question OpenBSD 6.7-current (ran sysupgrade Aug 12)
it is currently working using trunk(4)
po32 is the box in question not working using aggr(4)
Strange that the old box looks fine in the summary and the new box
looks broken even though trunk(4) still works.
> on the switch you are lagging with ?
>
> can you also do a
> sho run int port-channel ?
Just as a comparison here are all three.
OpenBSD 6.4 using trunk(4)
# sh run int port-channel 21
interface port-channel21
description Hades1 In
switchport
switchport mode trunk
OpenBSD 6.7-current using trunk(4)
# sh run int port-channel 33
interface port-channel33
description Styx2 Out
switchport
switchport access vlan 100
spanning-tree port type edge
OpenBSD 6.7-current using aggr(4)
# sh run int port-channel 32
interface port-channel32
description Styx2 In
switchport
switchport mode trunk
> sho run int ethernet x,y
Again, all of the interfaces for comparison
OpenBSD 6.4 using trunk(4)
# sh run int eth5/41
interface Ethernet5/41
description 3F-P19 Hades1 Inside
switchport
switchport mode trunk
channel-group 21 mode active
# sh run int eth6/41
interface Ethernet6/41
description 3F-P20 Hades1 Inside
switchport
switchport mode trunk
spanning-tree port type edge
channel-group 21 mode active
OpenBSD 6.7-current using trunk(4)
# sh run int eth7/38
interface Ethernet7/38
description 2F-P22 Styx2-Out
switchport
switchport access vlan 100
spanning-tree port type edge
channel-group 33
# sh run int eth8/38
interface Ethernet8/38
description 2F-P46 Styx2-Out
switchport
switchport access vlan 100
spanning-tree port type edge
channel-group 33
OpenBSD 6.7-current using aggr(4)
# sh run int eth5/38
interface Ethernet5/38
description 2F-P21 Styx2-In
switchport
switchport mode trunk
channel-group 32
# sh run int eth6/38
interface Ethernet6/38
description 2F-P45 Styx2-In
switchport
switchport mode trunk
channel-group 32
You may have noticed that the aggr(4) interfaces do not have
"spanning-tree port type edge" but that makes no difference
in the outcome. In fact, it is not supposed to work on a
trunked interface. From the switch:
# config t
# int eth5/38
# spanning-tree port type edge
Edge port type (portfast) should only be enabled on ports connected to a single
host. Connecting hubs, concentrators, switches, bridges, etc... to this
interface when edge port type (portfast) is enabled, can cause temporary
bridging loops.
Use with CAUTION
Edge Port Type (Portfast) has been configured on Ethernet5/38 but will only
have effect when the interface is in a non-trunking mode.
Thanks.
Winfred
> where x and y are the interfaces on the switch are in the LACP Port Channel
> ?
>
> Thanks
>
>
>
> On Wed, 12 Aug 2020 at 14:40, Winfred Harrelson <wharr...@kettering.edu>
> wrote:
>
> > On Tue, Aug 11, 2020 at 10:13:41PM +0200, Remi Locherer wrote:
> > > On Tue, Aug 11, 2020 at 02:07:32PM -0400, Winfred Harrelson wrote:
> > > > I know others are using the new aggr(4) interface but I am having a
> > > > problem with trying to use it on some new servers I have recently
> > > > gotten. Hoping I could get some help from someone here since my
> > > > searches have not been very fruitful.
> > > >
> > > > First off this is on a Supermicro X11DPi-N(T) and it is running a 6.7
> > > > snapshot from today because the 6.7 release hangs on trying to install.
> > > >
> > > > I have two Intel duel port XXV710 cards with SPF28 and trying to
> > > > create an LACP bond. Works fine using the trunk(4) interface but
> > > > not the aggr(4) interface. This is what I get:
> > > >
> > > > styx# ifconfig ixl0
> > > > ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr fe:e1:ba:d0:cc:aa
> > > > index 1 priority 0 llprio 3
> > > > trunk: trunkdev aggr1
> > > ^^^^^
> > > ixl0 is already member of aggr1
> > >
> > > > media: Ethernet autoselect (25GbaseSR full-duplex)
> > > > status: active
> > > > styx# ifconfig ixl2
> > > > ixl2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > > > lladdr fe:e1:ba:d0:cc:aa
> > > > index 3 priority 0 llprio 3
> > > > trunk: trunkdev aggr1
> > > ^^^^^
> > > same here
> >
> > Thanks for the quick reply.
> >
> > Unfortunately this is not the issue. I had tried several attempts
> > before sending the email and had forgotten to clean up first before
> > sending the examples. If the interface is already part of another
> > trunk group you will get an error attempting to add a different one.
> >
> > I did a sysupgrade this morning in case something had changed and
> > also updated the XXV710 cards to the newest firmware but with no luck.
> >
> >
> >
> > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl
> > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW
> > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28
> > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW
> > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29
> > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW
> > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0
> > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW
> > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1
> > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW
> > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2
> > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW
> > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3
> >
> > Yup, latest firmware.
> >
> >
> > wharrels@styx2:/etc# ifconfig ixl0
> > ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > lladdr 3c:fd:fe:ed:b7:28
> > index 1 priority 0 llprio 3
> > media: Ethernet autoselect (25GbaseSR full-duplex)
> > status: active
> > wharrels@styx2:/etc# ifconfig ixl2
> > ixl2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > lladdr 3c:fd:fe:eb:19:b0
> > index 3 priority 0 llprio 3
> > media: Ethernet autoselect (25GbaseSR full-duplex)
> > status: active
> > wharrels@styx2:/etc# ifconfig aggr1 create
> > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0
> > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2
> > wharrels@styx2:/etc# ifconfig aggr1 up
> > wharrels@styx2:/etc# ifconfig aggr1
> > aggr1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
> > lladdr fe:e1:ba:d0:7c:e9
> > index 11 priority 0 llprio 7
> > trunk: trunkproto lacp
> > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,0000,0000),
> > (0000,00:00:00:00:00:00,0000,0000,0000)]
> > ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9,
> > key 0xb, port pri 0x8000 number 0x1
> > ixl0 lacp actor state activity,aggregation,defaulted
> > ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00,
> > key 0x0, port pri 0x0 number 0x0
> > ixl0 lacp partner state activity,aggregation,sync
> > ixl0 port
> > ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9,
> > key 0xb, port pri 0x8000 number 0x3
> > ixl2 lacp actor state activity,aggregation,defaulted
> > ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00,
> > key 0x0, port pri 0x0 number 0x0
> > ixl2 lacp partner state activity,aggregation,sync
> > ixl2 port
> > groups: aggr
> > media: Ethernet autoselect
> > status: no carrier
> >
> >
> > Same issue. Anything else to try?
> >
> > This does work fine using trunk(4).
> >
> > Winfred
> >
> >
>
> --
> Kindest regards,
> Tom Smyth.
