Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Thu, Aug 20, 2020 at 07:31:15AM -0400, Winfred Harrelson wrote: > On Wed, Aug 19, 2020 at 10:29:33AM +1000, Aaron Mason wrote: > > On Wed, Aug 19, 2020 at 12:00 AM Winfred Harrelson > > wrote: > > > > > > On Tue, Aug 18, 2020 at 04:53:42PM +1000, Jonathan Matthew wrote: > > > > > > > > This sounds like multicast filters aren't working properly with your > > > > nic. > > > > trunk(4) puts trunk ports in promisc mode, so multicast filters don't > > > > matter, > > > > but aggr(4) doesn't. Could you try running 'tcpdump -ni ixl0' for a > > > > while and > > > > see if that side of the aggr starts working? > > > > > > I left the tcpdump running for a little over 5 minutes but that changed > > > nothing: > > > > > > > Did anything come up on the tcpdump while it was running? Maybe > > there's a clue there. > > Yes just what you would expect while on a trunk port. If that includes incoming LACP packets from the switch, can you capture a few and send them to me (directly, not to the list)?
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Wed, Aug 19, 2020 at 10:29:33AM +1000, Aaron Mason wrote: > On Wed, Aug 19, 2020 at 12:00 AM Winfred Harrelson > wrote: > > > > On Tue, Aug 18, 2020 at 04:53:42PM +1000, Jonathan Matthew wrote: > > > > > > This sounds like multicast filters aren't working properly with your nic. > > > trunk(4) puts trunk ports in promisc mode, so multicast filters don't > > > matter, > > > but aggr(4) doesn't. Could you try running 'tcpdump -ni ixl0' for a > > > while and > > > see if that side of the aggr starts working? > > > > I left the tcpdump running for a little over 5 minutes but that changed > > nothing: > > > > Did anything come up on the tcpdump while it was running? Maybe > there's a clue there. Yes just what you would expect while on a trunk port. Winfred
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Wed, Aug 19, 2020 at 12:00 AM Winfred Harrelson wrote: > > On Tue, Aug 18, 2020 at 04:53:42PM +1000, Jonathan Matthew wrote: > > > > This sounds like multicast filters aren't working properly with your nic. > > trunk(4) puts trunk ports in promisc mode, so multicast filters don't > > matter, > > but aggr(4) doesn't. Could you try running 'tcpdump -ni ixl0' for a while > > and > > see if that side of the aggr starts working? > > I left the tcpdump running for a little over 5 minutes but that changed > nothing: > Did anything come up on the tcpdump while it was running? Maybe there's a clue there. -- Aaron Mason - Programmer, open source addict I've taken my software vows - for beta or for worse
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Tue, Aug 18, 2020 at 04:53:42PM +1000, Jonathan Matthew wrote: > On Mon, Aug 17, 2020 at 03:32:35PM -0400, Winfred Harrelson wrote: > > On Mon, Aug 17, 2020 at 03:40:47PM +0200, Hrvoje Popovski wrote: > > > On 17.8.2020. 11:46, Stuart Henderson wrote: > > > > On 2020-08-15, Hrvoje Popovski wrote: > > > >> On 15.8.2020. 0:48, Hrvoje Popovski wrote: > > > >>> On 12.8.2020. 15:18, Winfred Harrelson wrote: > > > On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: > > > > Hi Winfred, > > > > the intel 710 is a complex card, I would suggest that you try > > > > updating the > > > > firmware on the card, available from intel.com or your card vendor, > > > > you may have to boot to a live linux cd to apply the firmware > > > > update, > > > > > > > > but I had some issues with the Intel XL710 cards and I had to > > > > update the > > > > firmware to get it working stable, > > > > > > > > I hope this helps > > > > Tom Smyth > > > > > > Adding misc@openbsd.org back to the CC for the record. > > > > > > Thanks for the quick reply. I didn't reply back yesterday because I > > > was having trouble getting the firmware updated from a Linux boot > > > disk. > > > I ended up having to try from a Windows boot disk. Unfortunately, I > > > am getting the same thing again: > > > > > > > > > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl > > > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, > > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 > > > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, > > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 > > > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, > > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 > > > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, > > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 > > > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port > > > 0, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 > > > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port > > > 1, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 > > > > > > Yup, all the XXV710 cards have been updated to newest firmware. > > > > > > Now for the (failed) attempt: > > > > > > wharrels@styx2:/etc# ifconfig ixl0 > > > ixl0: flags=8843 mtu 1500 > > > lladdr 3c:fd:fe:ed:b7:28 > > > index 1 priority 0 llprio 3 > > > media: Ethernet autoselect (25GbaseSR full-duplex) > > > status: active > > > wharrels@styx2:/etc# ifconfig ixl2 > > > ixl2: flags=8843 mtu 1500 > > > lladdr 3c:fd:fe:eb:19:b0 > > > index 3 priority 0 llprio 3 > > > media: Ethernet autoselect (25GbaseSR full-duplex) > > > status: active > > > wharrels@styx2:/etc# ifconfig aggr1 create > > > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 > > > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 > > > wharrels@styx2:/etc# ifconfig aggr1 up > > > wharrels@styx2:/etc# ifconfig aggr1 > > > aggr1: flags=8843 mtu 1500 > > > lladdr fe:e1:ba:d0:7c:e9 > > > index 11 priority 0 llprio 7 > > > trunk: trunkproto lacp > > > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), > > > (,00:00:00:00:00:00,,,)] > > > ixl0 lacp actor system pri 0x8000 mac > > > fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x1 > > > ixl0 lacp actor state activity,aggregation,defaulted > > > ixl0 lacp partner system pri 0x0 mac > > > 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 > > > ixl0 lacp partner state activity,aggregation,sync > > > ixl0 port > > > ixl2 lacp actor system pri 0x8000 mac > > > fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x3 > > > ixl2 lacp actor state activity,aggregation,defaulted > > > ixl2 lacp partner system pri 0x0 mac > > > 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 > > > ixl2 lacp partner state activity,aggregation,sync > > > ixl2 port > > > groups: aggr > > > media: Ethernet autoselect > > > status: no carrier > > > > > > > > > > > > I tried doing another sysupgrade this morning just in case something > > > had changed overnight but no luck. Any other ideas? > > > > > > Winfred > > > > > > >>> > > > >>> Hi, > > > >>> > > > >>>
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Mon, Aug 17, 2020 at 03:32:35PM -0400, Winfred Harrelson wrote: > On Mon, Aug 17, 2020 at 03:40:47PM +0200, Hrvoje Popovski wrote: > > On 17.8.2020. 11:46, Stuart Henderson wrote: > > > On 2020-08-15, Hrvoje Popovski wrote: > > >> On 15.8.2020. 0:48, Hrvoje Popovski wrote: > > >>> On 12.8.2020. 15:18, Winfred Harrelson wrote: > > On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: > > > Hi Winfred, > > > the intel 710 is a complex card, I would suggest that you try > > > updating the > > > firmware on the card, available from intel.com or your card vendor, > > > you may have to boot to a live linux cd to apply the firmware update, > > > > > > but I had some issues with the Intel XL710 cards and I had to update > > > the > > > firmware to get it working stable, > > > > > > I hope this helps > > > Tom Smyth > > > > Adding misc@openbsd.org back to the CC for the record. > > > > Thanks for the quick reply. I didn't reply back yesterday because I > > was having trouble getting the firmware updated from a Linux boot disk. > > I ended up having to try from a Windows boot disk. Unfortunately, I > > am getting the same thing again: > > > > > > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl > > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 > > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 > > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 > > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, > > FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 > > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port > > 0, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 > > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port > > 1, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 > > > > Yup, all the XXV710 cards have been updated to newest firmware. > > > > Now for the (failed) attempt: > > > > wharrels@styx2:/etc# ifconfig ixl0 > > ixl0: flags=8843 mtu 1500 > > lladdr 3c:fd:fe:ed:b7:28 > > index 1 priority 0 llprio 3 > > media: Ethernet autoselect (25GbaseSR full-duplex) > > status: active > > wharrels@styx2:/etc# ifconfig ixl2 > > ixl2: flags=8843 mtu 1500 > > lladdr 3c:fd:fe:eb:19:b0 > > index 3 priority 0 llprio 3 > > media: Ethernet autoselect (25GbaseSR full-duplex) > > status: active > > wharrels@styx2:/etc# ifconfig aggr1 create > > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 > > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 > > wharrels@styx2:/etc# ifconfig aggr1 up > > wharrels@styx2:/etc# ifconfig aggr1 > > aggr1: flags=8843 mtu 1500 > > lladdr fe:e1:ba:d0:7c:e9 > > index 11 priority 0 llprio 7 > > trunk: trunkproto lacp > > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), > > (,00:00:00:00:00:00,,,)] > > ixl0 lacp actor system pri 0x8000 mac > > fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x1 > > ixl0 lacp actor state activity,aggregation,defaulted > > ixl0 lacp partner system pri 0x0 mac > > 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 > > ixl0 lacp partner state activity,aggregation,sync > > ixl0 port > > ixl2 lacp actor system pri 0x8000 mac > > fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x3 > > ixl2 lacp actor state activity,aggregation,defaulted > > ixl2 lacp partner system pri 0x0 mac > > 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 > > ixl2 lacp partner state activity,aggregation,sync > > ixl2 port > > groups: aggr > > media: Ethernet autoselect > > status: no carrier > > > > > > > > I tried doing another sysupgrade this morning just in case something > > had changed overnight but no luck. Any other ideas? > > > > Winfred > > > > >>> > > >>> Hi, > > >>> > > >>> could you try install snapshot from http://ftp.hostserver.de/archive/ > > >>> that is older than Thu Jun 25 06:41:38 2020 UTC ... > > >>> > > >>> maybe this commit broke xxv710 > > >>> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup > > >>>
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Mon, Aug 17, 2020 at 03:40:47PM +0200, Hrvoje Popovski wrote: > On 17.8.2020. 11:46, Stuart Henderson wrote: > > On 2020-08-15, Hrvoje Popovski wrote: > >> On 15.8.2020. 0:48, Hrvoje Popovski wrote: > >>> On 12.8.2020. 15:18, Winfred Harrelson wrote: > On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: > > Hi Winfred, > > the intel 710 is a complex card, I would suggest that you try updating > > the > > firmware on the card, available from intel.com or your card vendor, > > you may have to boot to a live linux cd to apply the firmware update, > > > > but I had some issues with the Intel XL710 cards and I had to update the > > firmware to get it working stable, > > > > I hope this helps > > Tom Smyth > > Adding misc@openbsd.org back to the CC for the record. > > Thanks for the quick reply. I didn't reply back yesterday because I > was having trouble getting the firmware updated from a Linux boot disk. > I ended up having to try from a Windows boot disk. Unfortunately, I > am getting the same thing again: > > > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, > FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, > FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 > > Yup, all the XXV710 cards have been updated to newest firmware. > > Now for the (failed) attempt: > > wharrels@styx2:/etc# ifconfig ixl0 > ixl0: flags=8843 mtu 1500 > lladdr 3c:fd:fe:ed:b7:28 > index 1 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig ixl2 > ixl2: flags=8843 mtu 1500 > lladdr 3c:fd:fe:eb:19:b0 > index 3 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig aggr1 create > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 > wharrels@styx2:/etc# ifconfig aggr1 up > wharrels@styx2:/etc# ifconfig aggr1 > aggr1: flags=8843 mtu 1500 > lladdr fe:e1:ba:d0:7c:e9 > index 11 priority 0 llprio 7 > trunk: trunkproto lacp > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), > (,00:00:00:00:00:00,,,)] > ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, > key 0xb, port pri 0x8000 number 0x1 > ixl0 lacp actor state activity,aggregation,defaulted > ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, > key 0x0, port pri 0x0 number 0x0 > ixl0 lacp partner state activity,aggregation,sync > ixl0 port > ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, > key 0xb, port pri 0x8000 number 0x3 > ixl2 lacp actor state activity,aggregation,defaulted > ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, > key 0x0, port pri 0x0 number 0x0 > ixl2 lacp partner state activity,aggregation,sync > ixl2 port > groups: aggr > media: Ethernet autoselect > status: no carrier > > > > I tried doing another sysupgrade this morning just in case something > had changed overnight but no luck. Any other ideas? > > Winfred > > >>> > >>> Hi, > >>> > >>> could you try install snapshot from http://ftp.hostserver.de/archive/ > >>> that is older than Thu Jun 25 06:41:38 2020 UTC ... > >>> > >>> maybe this commit broke xxv710 > >>> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup > >>> > >>> i have vlans over aggr over x710-da2 with latest snapshot and it's > >>> working as expected .. > >>> > >>> ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW > >>> 7.3.60988 API 1.10, msix, 8 queues > >>> ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+"
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On 17.8.2020. 11:46, Stuart Henderson wrote: > On 2020-08-15, Hrvoje Popovski wrote: >> On 15.8.2020. 0:48, Hrvoje Popovski wrote: >>> On 12.8.2020. 15:18, Winfred Harrelson wrote: On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: > Hi Winfred, > the intel 710 is a complex card, I would suggest that you try updating > the > firmware on the card, available from intel.com or your card vendor, > you may have to boot to a live linux cd to apply the firmware update, > > but I had some issues with the Intel XL710 cards and I had to update the > firmware to get it working stable, > > I hope this helps > Tom Smyth Adding misc@openbsd.org back to the CC for the record. Thanks for the quick reply. I didn't reply back yesterday because I was having trouble getting the firmware updated from a Linux boot disk. I ended up having to try from a Windows boot disk. Unfortunately, I am getting the same thing again: wharrels@styx2:/home/wharrels# dmesg | grep ^ixl ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 Yup, all the XXV710 cards have been updated to newest firmware. Now for the (failed) attempt: wharrels@styx2:/etc# ifconfig ixl0 ixl0: flags=8843 mtu 1500 lladdr 3c:fd:fe:ed:b7:28 index 1 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig ixl2 ixl2: flags=8843 mtu 1500 lladdr 3c:fd:fe:eb:19:b0 index 3 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig aggr1 create wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 wharrels@styx2:/etc# ifconfig aggr1 up wharrels@styx2:/etc# ifconfig aggr1 aggr1: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:7c:e9 index 11 priority 0 llprio 7 trunk: trunkproto lacp trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), (,00:00:00:00:00:00,,,)] ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x1 ixl0 lacp actor state activity,aggregation,defaulted ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl0 lacp partner state activity,aggregation,sync ixl0 port ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x3 ixl2 lacp actor state activity,aggregation,defaulted ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl2 lacp partner state activity,aggregation,sync ixl2 port groups: aggr media: Ethernet autoselect status: no carrier I tried doing another sysupgrade this morning just in case something had changed overnight but no luck. Any other ideas? Winfred >>> >>> Hi, >>> >>> could you try install snapshot from http://ftp.hostserver.de/archive/ >>> that is older than Thu Jun 25 06:41:38 2020 UTC ... >>> >>> maybe this commit broke xxv710 >>> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup >>> >>> i have vlans over aggr over x710-da2 with latest snapshot and it's >>> working as expected .. >>> >>> ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW >>> 7.3.60988 API 1.10, msix, 8 queues >>> ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW >>> 7.3.60988 API 1.10, msix, 8 queues >>> >> >> with new firmware aggr is working >> >> ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW >> 8.0.61820 API 1.11, msix, 8 queues >> ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02:
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On 2020-08-15, Hrvoje Popovski wrote: > On 15.8.2020. 0:48, Hrvoje Popovski wrote: >> On 12.8.2020. 15:18, Winfred Harrelson wrote: >>> On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: Hi Winfred, the intel 710 is a complex card, I would suggest that you try updating the firmware on the card, available from intel.com or your card vendor, you may have to boot to a live linux cd to apply the firmware update, but I had some issues with the Intel XL710 cards and I had to update the firmware to get it working stable, I hope this helps Tom Smyth >>> >>> Adding misc@openbsd.org back to the CC for the record. >>> >>> Thanks for the quick reply. I didn't reply back yesterday because I >>> was having trouble getting the firmware updated from a Linux boot disk. >>> I ended up having to try from a Windows boot disk. Unfortunately, I >>> am getting the same thing again: >>> >>> >>> wharrels@styx2:/home/wharrels# dmesg | grep ^ixl >>> ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW >>> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 >>> ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW >>> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 >>> ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW >>> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 >>> ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW >>> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 >>> ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW >>> 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 >>> ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW >>> 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 >>> >>> Yup, all the XXV710 cards have been updated to newest firmware. >>> >>> Now for the (failed) attempt: >>> >>> wharrels@styx2:/etc# ifconfig ixl0 >>> ixl0: flags=8843 mtu 1500 >>> lladdr 3c:fd:fe:ed:b7:28 >>> index 1 priority 0 llprio 3 >>> media: Ethernet autoselect (25GbaseSR full-duplex) >>> status: active >>> wharrels@styx2:/etc# ifconfig ixl2 >>> ixl2: flags=8843 mtu 1500 >>> lladdr 3c:fd:fe:eb:19:b0 >>> index 3 priority 0 llprio 3 >>> media: Ethernet autoselect (25GbaseSR full-duplex) >>> status: active >>> wharrels@styx2:/etc# ifconfig aggr1 create >>> wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 >>> wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 >>> wharrels@styx2:/etc# ifconfig aggr1 up >>> wharrels@styx2:/etc# ifconfig aggr1 >>> aggr1: flags=8843 mtu 1500 >>> lladdr fe:e1:ba:d0:7c:e9 >>> index 11 priority 0 llprio 7 >>> trunk: trunkproto lacp >>> trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), >>> (,00:00:00:00:00:00,,,)] >>> ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, >>> key 0xb, port pri 0x8000 number 0x1 >>> ixl0 lacp actor state activity,aggregation,defaulted >>> ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key >>> 0x0, port pri 0x0 number 0x0 >>> ixl0 lacp partner state activity,aggregation,sync >>> ixl0 port >>> ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, >>> key 0xb, port pri 0x8000 number 0x3 >>> ixl2 lacp actor state activity,aggregation,defaulted >>> ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key >>> 0x0, port pri 0x0 number 0x0 >>> ixl2 lacp partner state activity,aggregation,sync >>> ixl2 port >>> groups: aggr >>> media: Ethernet autoselect >>> status: no carrier >>> >>> >>> >>> I tried doing another sysupgrade this morning just in case something >>> had changed overnight but no luck. Any other ideas? >>> >>> Winfred >>> >> >> Hi, >> >> could you try install snapshot from http://ftp.hostserver.de/archive/ >> that is older than Thu Jun 25 06:41:38 2020 UTC ... >> >> maybe this commit broke xxv710 >> http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup >> >> i have vlans over aggr over x710-da2 with latest snapshot and it's >> working as expected .. >> >> ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW >> 7.3.60988 API 1.10, msix, 8 queues >> ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW >> 7.3.60988 API 1.10, msix, 8 queues >> > > with new firmware aggr is working > > ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues > ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues That's the same firmware as in your previous (failing) report, so is that "with new firmware and a snapshot
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On 15.8.2020. 0:48, Hrvoje Popovski wrote: > On 12.8.2020. 15:18, Winfred Harrelson wrote: >> On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: >>> Hi Winfred, >>> the intel 710 is a complex card, I would suggest that you try updating the >>> firmware on the card, available from intel.com or your card vendor, >>> you may have to boot to a live linux cd to apply the firmware update, >>> >>> but I had some issues with the Intel XL710 cards and I had to update the >>> firmware to get it working stable, >>> >>> I hope this helps >>> Tom Smyth >> >> Adding misc@openbsd.org back to the CC for the record. >> >> Thanks for the quick reply. I didn't reply back yesterday because I >> was having trouble getting the firmware updated from a Linux boot disk. >> I ended up having to try from a Windows boot disk. Unfortunately, I >> am getting the same thing again: >> >> >> wharrels@styx2:/home/wharrels# dmesg | grep ^ixl >> ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW >> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 >> ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW >> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 >> ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW >> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 >> ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW >> 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 >> ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW >> 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 >> ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW >> 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 >> >> Yup, all the XXV710 cards have been updated to newest firmware. >> >> Now for the (failed) attempt: >> >> wharrels@styx2:/etc# ifconfig ixl0 >> ixl0: flags=8843 mtu 1500 >> lladdr 3c:fd:fe:ed:b7:28 >> index 1 priority 0 llprio 3 >> media: Ethernet autoselect (25GbaseSR full-duplex) >> status: active >> wharrels@styx2:/etc# ifconfig ixl2 >> ixl2: flags=8843 mtu 1500 >> lladdr 3c:fd:fe:eb:19:b0 >> index 3 priority 0 llprio 3 >> media: Ethernet autoselect (25GbaseSR full-duplex) >> status: active >> wharrels@styx2:/etc# ifconfig aggr1 create >> wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 >> wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 >> wharrels@styx2:/etc# ifconfig aggr1 up >> wharrels@styx2:/etc# ifconfig aggr1 >> aggr1: flags=8843 mtu 1500 >> lladdr fe:e1:ba:d0:7c:e9 >> index 11 priority 0 llprio 7 >> trunk: trunkproto lacp >> trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), >> (,00:00:00:00:00:00,,,)] >> ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key >> 0xb, port pri 0x8000 number 0x1 >> ixl0 lacp actor state activity,aggregation,defaulted >> ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key >> 0x0, port pri 0x0 number 0x0 >> ixl0 lacp partner state activity,aggregation,sync >> ixl0 port >> ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key >> 0xb, port pri 0x8000 number 0x3 >> ixl2 lacp actor state activity,aggregation,defaulted >> ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key >> 0x0, port pri 0x0 number 0x0 >> ixl2 lacp partner state activity,aggregation,sync >> ixl2 port >> groups: aggr >> media: Ethernet autoselect >> status: no carrier >> >> >> >> I tried doing another sysupgrade this morning just in case something >> had changed overnight but no luck. Any other ideas? >> >> Winfred >> > > Hi, > > could you try install snapshot from http://ftp.hostserver.de/archive/ > that is older than Thu Jun 25 06:41:38 2020 UTC ... > > maybe this commit broke xxv710 > http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup > > i have vlans over aggr over x710-da2 with latest snapshot and it's > working as expected .. > > ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW > 7.3.60988 API 1.10, msix, 8 queues > ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW > 7.3.60988 API 1.10, msix, 8 queues > with new firmware aggr is working ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On 12.8.2020. 15:18, Winfred Harrelson wrote: > On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: >> Hi Winfred, >> the intel 710 is a complex card, I would suggest that you try updating the >> firmware on the card, available from intel.com or your card vendor, >> you may have to boot to a live linux cd to apply the firmware update, >> >> but I had some issues with the Intel XL710 cards and I had to update the >> firmware to get it working stable, >> >> I hope this helps >> Tom Smyth > > Adding misc@openbsd.org back to the CC for the record. > > Thanks for the quick reply. I didn't reply back yesterday because I > was having trouble getting the firmware updated from a Linux boot disk. > I ended up having to try from a Windows boot disk. Unfortunately, I > am getting the same thing again: > > > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 > > Yup, all the XXV710 cards have been updated to newest firmware. > > Now for the (failed) attempt: > > wharrels@styx2:/etc# ifconfig ixl0 > ixl0: flags=8843 mtu 1500 > lladdr 3c:fd:fe:ed:b7:28 > index 1 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig ixl2 > ixl2: flags=8843 mtu 1500 > lladdr 3c:fd:fe:eb:19:b0 > index 3 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig aggr1 create > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 > wharrels@styx2:/etc# ifconfig aggr1 up > wharrels@styx2:/etc# ifconfig aggr1 > aggr1: flags=8843 mtu 1500 > lladdr fe:e1:ba:d0:7c:e9 > index 11 priority 0 llprio 7 > trunk: trunkproto lacp > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), > (,00:00:00:00:00:00,,,)] > ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key > 0xb, port pri 0x8000 number 0x1 > ixl0 lacp actor state activity,aggregation,defaulted > ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key > 0x0, port pri 0x0 number 0x0 > ixl0 lacp partner state activity,aggregation,sync > ixl0 port > ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key > 0xb, port pri 0x8000 number 0x3 > ixl2 lacp actor state activity,aggregation,defaulted > ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key > 0x0, port pri 0x0 number 0x0 > ixl2 lacp partner state activity,aggregation,sync > ixl2 port > groups: aggr > media: Ethernet autoselect > status: no carrier > > > > I tried doing another sysupgrade this morning just in case something > had changed overnight but no luck. Any other ideas? > > Winfred > Hi, could you try install snapshot from http://ftp.hostserver.de/archive/ that is older than Thu Jun 25 06:41:38 2020 UTC ... maybe this commit broke xxv710 http://cvsweb.openbsd.org/cgi-bin/cvsweb/src/sys/dev/pci/if_ixl.c?rev=1.56=text/x-cvsweb-markup i have vlans over aggr over x710-da2 with latest snapshot and it's working as expected .. ixl0 at pci1 dev 0 function 0 "Intel X710 SFP+" rev 0x02: port 0, FW 7.3.60988 API 1.10, msix, 8 queues ixl1 at pci1 dev 0 function 1 "Intel X710 SFP+" rev 0x02: port 1, FW 7.3.60988 API 1.10, msix, 8 queues could you send output from these two commands .. ifconfig ixl sff ifconfig ixl media
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Wed, Aug 12, 2020 at 02:54:36PM +0100, Tom Smyth wrote: > What is the Switch telling you about the LACP ? This is a Cisco switch running Nexus not IOS so I am not quite as familiar with it as the IOS ones so I may have a few things wrong here. Also, you are asking for a lot of into so this is a long email. This is what I think you are asking for. > can you do a > > show port-channel # sh port-channel ^ % Incomplete command at '^' marker This is not right for NXOS. Do you mean this: # show port-channel database ... port-channel32 Last membership update is successful 2 ports in total, 2 ports up First operational port is Ethernet5/38 Age of the port-channel is 5d:01h:54m:27s Time since last bundle is 5d:01h:50m:54s Last bundled member is Ethernet6/38 Ports: Ethernet5/38[on] [up] * Ethernet6/38[on] [up] port-channel33 Last membership update is successful 2 ports in total, 2 ports up First operational port is Ethernet7/38 Age of the port-channel is 5d:01h:53m:58s Time since last bundle is 5d:01h:48m:28s Last bundled member is Ethernet8/38 Ports: Ethernet7/38[on] [up] * Ethernet8/38[on] [up] > show port-channel detailed # sh port-channel detailed ^ % Invalid command at '^' marker. Do you mean this: show port-channel internal event-history all | include port-channel32 next 20 port-channel32 channel: 32 bundle : 65535 ifindex: 0x161f admin mode : on oper mode : on nports : 2 active : 2 pre cfg: 0 ltl: 0x15 (21) lif: 0x12bc iod: 0x2bf (703) global id : 20 flag : 0 lock count : 0 num. of SIs: 0 ac mbrs: 0 0 lacp graceful conv disable : 0 lacp suspend indiv disable : 0 pc min-links : 1 pc max-bundle: 32 -- PC SDB: Is a member of port-channel32 FSM: has 191 logged transitions< 1) Event:ESQ_RSP length:38, at 75399 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_MOD_MEMB_LTL(69637) RRtoken:0x4C0F7380 2) Event:ESQ_REQ length:38, at 75466 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_TX] Dst:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497) RRtoken:0x4C0F73D9 3) Event:ESQ_RSP length:38, at 75593 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A204A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497) RRtoken:0x4C0F73D9 4) Event:ESQ_REQ length:38, at 75646 usecs after Wed Aug 12 17:18:28 2020 -- PC SDB: Is a member of port-channel32 FSM: has 191 logged transitions< 1) Event:ESQ_RSP length:38, at 263607 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_PIXM_LOCAL(153), Opc:MTS_OPC_PIXM_MOD_MEMB_LTL(69637) RRtoken:0x4C0F7603 2) Event:ESQ_REQ length:38, at 263663 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_TX] Dst:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497) RRtoken:0x4C0F7613 3) Event:ESQ_RSP length:38, at 263825 usecs after Wed Aug 12 17:18:28 2020 Instance:0x1A284A00, Seq Id:0x1, Ret:SUCCESS [E_MTS_RX] Src:MTS_SAP_UFDM_SYS(193), Opc:MTS_OPC_PCM_PROTOCOL_DN(63497) RRtoken:0x4C0F7613 4) Event:ESQ_REQ length:38, at 263887 usecs after Wed Aug 12 17:18:28 2020 Or maybe this: # show port-channel internal info interface po32 port-channel32 channel: 32 bundle : 65535 ifindex: 0x161f admin mode : on oper mode : on nports : 2 active : 2 pre cfg: 0 ltl: 0x15 (21) lif: 0x12bc iod: 0x2bf (703) global id : 20 flag : 0 lock count : 0 num. of SIs: 0 ac mbrs: 0 0 lacp graceful conv disable : 0 lacp suspend indiv disable : 0 pc min-links : 1 pc max-bundle: 32 pc max active members: 32 pc is-suspend-minlinks : 0 port load defer enable : 0 port-channel bfd config enabled : 0 port-channel bfd config complete: 0 port-channel bfd destination: null port-channel bfd start timeout: 0 lacp fast-select-hot-standby disable : 0 port-channel port hash-distribution: none ethpm bundle lock count : 0 lacp delayed-enable fop Ethernet5/38 0x1a204a00 lacp delayed-enable : 0 lacp delayed-enable cfg-port none lacp delayed-enable
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
What is the Switch telling you about the LACP ? can you do a show port-channel show port-channel detailed sho port-channel summary on the switch you are lagging with ? can you also do a sho run int port-channel ? sho run int ethernet x,y where x and y are the interfaces on the switch are in the LACP Port Channel ? Thanks On Wed, 12 Aug 2020 at 14:40, Winfred Harrelson wrote: > On Tue, Aug 11, 2020 at 10:13:41PM +0200, Remi Locherer wrote: > > On Tue, Aug 11, 2020 at 02:07:32PM -0400, Winfred Harrelson wrote: > > > I know others are using the new aggr(4) interface but I am having a > > > problem with trying to use it on some new servers I have recently > > > gotten. Hoping I could get some help from someone here since my > > > searches have not been very fruitful. > > > > > > First off this is on a Supermicro X11DPi-N(T) and it is running a 6.7 > > > snapshot from today because the 6.7 release hangs on trying to install. > > > > > > I have two Intel duel port XXV710 cards with SPF28 and trying to > > > create an LACP bond. Works fine using the trunk(4) interface but > > > not the aggr(4) interface. This is what I get: > > > > > > styx# ifconfig ixl0 > > > ixl0: flags=8843 mtu 1500 > > > lladdr fe:e1:ba:d0:cc:aa > > > index 1 priority 0 llprio 3 > > > trunk: trunkdev aggr1 > > ^ > > ixl0 is already member of aggr1 > > > > > media: Ethernet autoselect (25GbaseSR full-duplex) > > > status: active > > > styx# ifconfig ixl2 > > > ixl2: flags=8843 mtu 1500 > > > lladdr fe:e1:ba:d0:cc:aa > > > index 3 priority 0 llprio 3 > > > trunk: trunkdev aggr1 > > ^ > > same here > > Thanks for the quick reply. > > Unfortunately this is not the issue. I had tried several attempts > before sending the email and had forgotten to clean up first before > sending the examples. If the interface is already part of another > trunk group you will get an error attempting to add a different one. > > I did a sysupgrade this morning in case something had changed and > also updated the XXV710 cards to the newest firmware but with no luck. > > > > wharrels@styx2:/home/wharrels# dmesg | grep ^ixl > ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 > ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 > ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 > ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW > 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 > ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 > ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW > 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 > > Yup, latest firmware. > > > wharrels@styx2:/etc# ifconfig ixl0 > ixl0: flags=8843 mtu 1500 > lladdr 3c:fd:fe:ed:b7:28 > index 1 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig ixl2 > ixl2: flags=8843 mtu 1500 > lladdr 3c:fd:fe:eb:19:b0 > index 3 priority 0 llprio 3 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > wharrels@styx2:/etc# ifconfig aggr1 create > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 > wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 > wharrels@styx2:/etc# ifconfig aggr1 up > wharrels@styx2:/etc# ifconfig aggr1 > aggr1: flags=8843 mtu 1500 > lladdr fe:e1:ba:d0:7c:e9 > index 11 priority 0 llprio 7 > trunk: trunkproto lacp > trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), > (,00:00:00:00:00:00,,,)] > ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, > key 0xb, port pri 0x8000 number 0x1 > ixl0 lacp actor state activity,aggregation,defaulted > ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, > key 0x0, port pri 0x0 number 0x0 > ixl0 lacp partner state activity,aggregation,sync > ixl0 port > ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, > key 0xb, port pri 0x8000 number 0x3 > ixl2 lacp actor state activity,aggregation,defaulted > ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, > key 0x0, port pri 0x0 number 0x0 > ixl2 lacp partner state activity,aggregation,sync > ixl2 port > groups: aggr > media: Ethernet autoselect > status: no carrier > > > Same issue. Anything else to try? > > This does work fine using trunk(4). > >
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Tue, Aug 11, 2020 at 10:13:41PM +0200, Remi Locherer wrote: > On Tue, Aug 11, 2020 at 02:07:32PM -0400, Winfred Harrelson wrote: > > I know others are using the new aggr(4) interface but I am having a > > problem with trying to use it on some new servers I have recently > > gotten. Hoping I could get some help from someone here since my > > searches have not been very fruitful. > > > > First off this is on a Supermicro X11DPi-N(T) and it is running a 6.7 > > snapshot from today because the 6.7 release hangs on trying to install. > > > > I have two Intel duel port XXV710 cards with SPF28 and trying to > > create an LACP bond. Works fine using the trunk(4) interface but > > not the aggr(4) interface. This is what I get: > > > > styx# ifconfig ixl0 > > ixl0: flags=8843 mtu 1500 > > lladdr fe:e1:ba:d0:cc:aa > > index 1 priority 0 llprio 3 > > trunk: trunkdev aggr1 > ^ > ixl0 is already member of aggr1 > > > media: Ethernet autoselect (25GbaseSR full-duplex) > > status: active > > styx# ifconfig ixl2 > > ixl2: flags=8843 mtu 1500 > > lladdr fe:e1:ba:d0:cc:aa > > index 3 priority 0 llprio 3 > > trunk: trunkdev aggr1 > ^ > same here Thanks for the quick reply. Unfortunately this is not the issue. I had tried several attempts before sending the email and had forgotten to clean up first before sending the examples. If the interface is already part of another trunk group you will get an error attempting to add a different one. I did a sysupgrade this morning in case something had changed and also updated the XXV710 cards to the newest firmware but with no luck. wharrels@styx2:/home/wharrels# dmesg | grep ^ixl ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 Yup, latest firmware. wharrels@styx2:/etc# ifconfig ixl0 ixl0: flags=8843 mtu 1500 lladdr 3c:fd:fe:ed:b7:28 index 1 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig ixl2 ixl2: flags=8843 mtu 1500 lladdr 3c:fd:fe:eb:19:b0 index 3 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig aggr1 create wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 wharrels@styx2:/etc# ifconfig aggr1 up wharrels@styx2:/etc# ifconfig aggr1 aggr1: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:7c:e9 index 11 priority 0 llprio 7 trunk: trunkproto lacp trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), (,00:00:00:00:00:00,,,)] ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x1 ixl0 lacp actor state activity,aggregation,defaulted ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl0 lacp partner state activity,aggregation,sync ixl0 port ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x3 ixl2 lacp actor state activity,aggregation,defaulted ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl2 lacp partner state activity,aggregation,sync ixl2 port groups: aggr media: Ethernet autoselect status: no carrier Same issue. Anything else to try? This does work fine using trunk(4). Winfred
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Tue, Aug 11, 2020 at 07:52:10PM +0100, Tom Smyth wrote: > Hi Winfred, > the intel 710 is a complex card, I would suggest that you try updating the > firmware on the card, available from intel.com or your card vendor, > you may have to boot to a live linux cd to apply the firmware update, > > but I had some issues with the Intel XL710 cards and I had to update the > firmware to get it working stable, > > I hope this helps > Tom Smyth Adding misc@openbsd.org back to the CC for the record. Thanks for the quick reply. I didn't reply back yesterday because I was having trouble getting the firmware updated from a Linux boot disk. I ended up having to try from a Windows boot disk. Unfortunately, I am getting the same thing again: wharrels@styx2:/home/wharrels# dmesg | grep ^ixl ixl0 at pci5 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:28 ixl1 at pci5 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:ed:b7:29 ixl2 at pci8 dev 0 function 0 "Intel XXV710 SFP28" rev 0x02: port 0, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b0 ixl3 at pci8 dev 0 function 1 "Intel XXV710 SFP28" rev 0x02: port 1, FW 8.0.61820 API 1.11, msix, 8 queues, address 3c:fd:fe:eb:19:b1 ixl4 at pci12 dev 0 function 0 "Intel X722 10GBASE-T" rev 0x09: port 0, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f2 ixl5 at pci12 dev 0 function 1 "Intel X722 10GBASE-T" rev 0x09: port 1, FW 3.1.57069 API 1.5, msix, 8 queues, address 3c:ec:ef:1a:df:f3 Yup, all the XXV710 cards have been updated to newest firmware. Now for the (failed) attempt: wharrels@styx2:/etc# ifconfig ixl0 ixl0: flags=8843 mtu 1500 lladdr 3c:fd:fe:ed:b7:28 index 1 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig ixl2 ixl2: flags=8843 mtu 1500 lladdr 3c:fd:fe:eb:19:b0 index 3 priority 0 llprio 3 media: Ethernet autoselect (25GbaseSR full-duplex) status: active wharrels@styx2:/etc# ifconfig aggr1 create wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl0 wharrels@styx2:/etc# ifconfig aggr1 trunkport ixl2 wharrels@styx2:/etc# ifconfig aggr1 up wharrels@styx2:/etc# ifconfig aggr1 aggr1: flags=8843 mtu 1500 lladdr fe:e1:ba:d0:7c:e9 index 11 priority 0 llprio 7 trunk: trunkproto lacp trunk id: [(8000,fe:e1:ba:d0:7c:e9,000B,,), (,00:00:00:00:00:00,,,)] ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x1 ixl0 lacp actor state activity,aggregation,defaulted ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl0 lacp partner state activity,aggregation,sync ixl0 port ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d0:7c:e9, key 0xb, port pri 0x8000 number 0x3 ixl2 lacp actor state activity,aggregation,defaulted ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key 0x0, port pri 0x0 number 0x0 ixl2 lacp partner state activity,aggregation,sync ixl2 port groups: aggr media: Ethernet autoselect status: no carrier I tried doing another sysupgrade this morning just in case something had changed overnight but no luck. Any other ideas? Winfred
Re: aggr(4) not working with Intel XXV710 SFP28 on a Supermicro X11DPi-N(T)
On Tue, Aug 11, 2020 at 02:07:32PM -0400, Winfred Harrelson wrote: > I know others are using the new aggr(4) interface but I am having a > problem with trying to use it on some new servers I have recently > gotten. Hoping I could get some help from someone here since my > searches have not been very fruitful. > > First off this is on a Supermicro X11DPi-N(T) and it is running a 6.7 > snapshot from today because the 6.7 release hangs on trying to install. > > I have two Intel duel port XXV710 cards with SPF28 and trying to > create an LACP bond. Works fine using the trunk(4) interface but > not the aggr(4) interface. This is what I get: > > styx# ifconfig ixl0 > ixl0: flags=8843 mtu 1500 > lladdr fe:e1:ba:d0:cc:aa > index 1 priority 0 llprio 3 > trunk: trunkdev aggr1 ^ ixl0 is already member of aggr1 > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > styx# ifconfig ixl2 > ixl2: flags=8843 mtu 1500 > lladdr fe:e1:ba:d0:cc:aa > index 3 priority 0 llprio 3 > trunk: trunkdev aggr1 ^ same here > media: Ethernet autoselect (25GbaseSR full-duplex) > status: active > > So everything looks good. Now to create the bond: > > styx# ifconfig aggr0 create > styx# ifconfig aggr0 trunkport ixl0 > styx# ifconfig aggr0 trunkport ixl2 > styx# ifconfig aggr0 up > here you add them to aggr0. have you checked if aggr1 is operational? > No error message so that seems good, however when I look at the bond > I get "no carrier": > > styx# ifconfig aggr0 > aggr0: flags=8843 mtu 1500 > lladdr fe:e1:ba:d1:7d:55 > index 13 priority 0 llprio 7 > trunk: trunkproto lacp > trunk id: [(8000,fe:e1:ba:d1:7d:55,000D,,), > (,00:00:00:00:00:00,,,)] > ixl0 lacp actor system pri 0x8000 mac fe:e1:ba:d1:7d:55, key > 0xd, port pri 0x8000 number 0x1 > ixl0 lacp actor state activity,aggregation,defaulted > ixl0 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key > 0x0, port pri 0x0 number 0x0 > ixl0 lacp partner state activity,aggregation,sync > ixl0 port > ixl2 lacp actor system pri 0x8000 mac fe:e1:ba:d1:7d:55, key > 0xd, port pri 0x8000 number 0x3 > ixl2 lacp actor state activity,aggregation,defaulted > ixl2 lacp partner system pri 0x0 mac 00:00:00:00:00:00, key > 0x0, port pri 0x0 number 0x0 > ixl2 lacp partner state activity,aggregation,sync > ixl2 port > groups: aggr > media: Ethernet autoselect > status: no carrier > > If I add debug to the ifconfig this is all I get in messages: > > Aug 11 13:45:37 styx /bsd: aggr0 ixl0 trunkport: creating port > Aug 11 13:45:37 styx /bsd: aggr0 ixl0 mux: BEGIN (BEGIN) -> DETACHED > Aug 11 13:45:37 styx /bsd: aggr0 ixl0 rxm: BEGIN (BEGIN) -> INITIALIZE > Aug 11 13:45:37 styx /bsd: aggr0 ixl0 rxm: INITIALIZE (UCT) -> PORT_DISABLED > Aug 11 13:45:37 styx /bsd: aggr0 ixl0 rxm: PORT_DISABLED (port_enabled) -> > LACP_DISABLED > Aug 11 13:45:44 styx /bsd: aggr0 ixl2 trunkport: creating port > Aug 11 13:45:44 styx /bsd: aggr0 ixl2 mux: BEGIN (BEGIN) -> DETACHED > Aug 11 13:45:44 styx /bsd: aggr0 ixl2 rxm: BEGIN (BEGIN) -> INITIALIZE > Aug 11 13:45:44 styx /bsd: aggr0 ixl2 rxm: INITIALIZE (UCT) -> PORT_DISABLED > Aug 11 13:45:44 styx /bsd: aggr0 ixl2 rxm: PORT_DISABLED (port_enabled) -> > LACP_DISABLED > Aug 11 13:45:48 styx /bsd: aggr0 ixl0 rxm: LACP_DISABLED (LACP_Enabled) -> > PORT_DISABLED > Aug 11 13:45:48 styx /bsd: aggr0 ixl0 rxm: PORT_DISABLED (port_enabled) -> > EXPIRED > Aug 11 13:45:48 styx /bsd: aggr0 ixl2 rxm: LACP_DISABLED (LACP_Enabled) -> > PORT_DISABLED > Aug 11 13:45:48 styx /bsd: aggr0 ixl2 rxm: PORT_DISABLED (port_enabled) -> > EXPIRED > Aug 11 13:45:51 styx /bsd: aggr0 ixl0 rxm: EXPIRED (current_while_timer > expired) -> DEFAULTED > Aug 11 13:45:51 styx /bsd: aggr0 ixl0: selection logic: unselected (rxm > !CURRENT) > Aug 11 13:45:51 styx /bsd: aggr0 ixl2 rxm: EXPIRED (current_while_timer > expired) -> DEFAULTED > Aug 11 13:45:51 styx /bsd: aggr0 ixl2: selection logic: unselected (rxm > !CURRENT) > > > Only thing I have found so far is a few threads from Nov - Dec 2019: > > https://marc.info/?l=openbsd-misc=157641049328878=2 > https://marc.info/?l=openbsd-misc=157537227501920=2 > https://marc.info/?l=openbsd-misc=157374699108084=2 > > > As with the above threads this works fine using the trunk(4) interface > just not the newer aggr(4) interface. > > They are connected to a Cisco Nexus 9508 switch. Any ideas of what > I can try? dmesg below > > Thanks > > Winfred > > > > stem subclass miscellaneous, rev 0x07) at pci4 dev 8 function 6 not configured > vendor "Intel", unknown product 0x208d (class system subclass miscellaneous, > rev 0x07) at
