Re: cxgbe firmware error on latest RELENG_13

Mon, 28 Feb 2022 11:43:57 -0800 

On 2/28/22 11:36 AM, mike tancsa wrote:

Hi Navdeep,
    I updated a pair of releng13 boxes today as I saw there are new drivers/firmware. However, the firmware seems to fail on both in different ways
t5nex0: <Chelsio T540-CR> mem 0x91300000-0x9137ffff,0x90000000-0x90ffffff,0x91984000-0x91985fff irq 17 at device 0.4 on pci2 
t5nex0: failed to load firmware module: 2. cfg 0, fw 0
t5nex0: firmware on card (1.25.0.40) is older than the version bundled with this driver, but the driver cannot take corrective action because it is unable to load the firmware module.
It looks like the firmware KLD isn't available even though you have t5fw_cfg_load="yes" in loader.conf.
Can you try removing the cxgbe lines from loader.conf temporarily and kldload if_cxgbe manually after boot? It should be able to find the KLD with the firmware and update the card. Then restore the lines in loader.conf and see if it comes up properly. 

Regards,
Navdeep


t5nex0: KLD with default config is not available.
t5nex0: failed to upload config file to card: 2.
t5nex0: failed (2) to configure card with "default" profile, will fall back to a basic configuration and retry. 
cxl0: <port 0> on t5nex0
cxl0: Ethernet address: 00:07:43:29:98:90
cxl0: 12 txq, 8 rxq (NIC)
cxl1: <port 1> on t5nex0
cxl1: Ethernet address: 00:07:43:29:98:98
cxl1: 12 txq, 8 rxq (NIC)
cxl2: <port 2> on t5nex0
cxl2: Ethernet address: 00:07:43:29:98:a0
cxl2: 12 txq, 8 rxq (NIC)
cxl3: <port 3> on t5nex0
cxl3: Ethernet address: 00:07:43:29:98:a8
cxl3: 12 txq, 8 rxq (NIC)
t5nex0: PCIe gen3 x8, 4 ports, 34 MSI-X interrupts, 84 eq, 33 iq

And on another adapter, different box
t5nex0: <Chelsio T520-LL-CR> mem 0xb2300000-0xb237ffff,0xb1000000-0xb1ffffff,0xb28a0000-0xb28a1fff irq 16 at device 0.4 on pci1 
t5nex0: command 0x4 in mbox 4 timed out (0x80000000).
t5nex0: mbox 4 cmdsent 04a0000000000001 0f41000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 t5nex0: mbox 4 current 04a0000000000001 0f41000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 0000000000000000 
t5nex0: firmware reports adapter error: Crash (0x80000000)
t5nex0: device log follows.
       Seq#           Tstamp     Level  Facility  Message
         0           507376      INFO      CORE  log initialized @ 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000          1           510262      INFO      CORE  bootstrap firmware took 605 msecs to run          2           511784      INFO      CORE  Serial Configuration version: 0x11101000 VPD version: 0x2          3           511785    NOTICE      CORE  pcie: Serial Configuration SPARE3 has Cookie Generation enabled.          4           511786      INFO      CORE  pcie: npf 7 (pfbitmap 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 
t5nex0: encountered fatal error, adapter stopped.
t5nex0: firmware reports adapter error: Crash (0x80000000)
t5nex0: device log follows.
       Seq#           Tstamp     Level  Facility  Message
         0           507376      INFO      CORE  log initialized @ 0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000          1           510262      INFO      CORE  bootstrap firmware took 605 msecs to run          2           511784      INFO      CORE  Serial Configuration version: 0x11101000 VPD version: 0x2          3           511785    NOTICE      CORE  pcie: Serial Configuration SPARE3 has Cookie Generation enabled.          4           511786      INFO      CORE  pcie: npf 7 (pfbitmap 0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4 
t5nex0: failed to connect to the firmware: 6, 0.  PCIE_FW 0x80000000
t5nex0: error during attach, adapter is now in recovery mode.
none0@pci0:1:0:5:       class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5511 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T520-LL-CR Unified Wire Storage Controller'
     class      = mass storage
     subclass   = SCSI
    bar   [10] = type Memory, range 64, base 0xb2280000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0xb2200000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0xb2890000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
                  max read 512
                  link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)
     cap 11[b0] = MSI-X supports 40 messages
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 0 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5811
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1
none1@pci0:1:0:6:       class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5611 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T520-LL-CR Unified Wire Storage Controller'
     class      = serial bus
     subclass   = Fibre Channel
    bar   [10] = type Memory, range 64, base 0xb2100000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0xb2080000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0xb2880000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
                  max read 512
                  link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)
     cap 11[b0] = MSI-X supports 40 messages
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 0 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5811
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1
t5iov3@pci0:2:0:3:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5003 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T540-CR Unified Wire Ethernet Controller'
     class      = network
     subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0x91480000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91400000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91986000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
                  max read 512
                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
     cap 11[b0] = MSI-X supports 8 messages
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 16 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5803
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1
t5nex0@pci0:2:0:4:      class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5403 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T540-CR Unified Wire Ethernet Controller'
     class      = network
     subclass   = ethernet
    bar   [10] = type Memory, range 64, base 0x91300000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x90000000, size 16777216, enabled     bar   [20] = type Memory, range 64, base 0x91984000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR
                  max read 4096
                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
     cap 11[b0] = MSI-X supports 128 messages, enabled
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 0 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5803
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1
none7@pci0:2:0:5:       class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5503 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T540-CR Unified Wire Storage Controller'
     class      = mass storage
     subclass   = SCSI
    bar   [10] = type Memory, range 64, base 0x91280000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91200000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91982000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
                  max read 512
                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
     cap 11[b0] = MSI-X supports 40 messages
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 0 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5803
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1
none8@pci0:2:0:6:       class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425 device=0x5603 subvendor=0x1425 subdevice=0x0000 
     vendor     = 'Chelsio Communications Inc'
     device     = 'T540-CR Unified Wire Storage Controller'
     class      = serial bus
     subclass   = Fibre Channel
    bar   [10] = type Memory, range 64, base 0x91100000, size 524288, enabled     bar   [18] = type Memory, range 64, base 0x91080000, size 524288, enabled     bar   [20] = type Memory, range 64, base 0x91980000, size 8192, enabled 
     cap 01[40] = powerspec 3  supports D0 D3  current D0
     cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
     cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
                  max read 512
                  link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
     cap 11[b0] = MSI-X supports 40 messages
                  Table in map 0x20[0x0], PBA in map 0x20[0x1000]
     cap 03[d0] = VPD
     ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
     ecap 0003[170] = Serial 1 0000000000000000
     ecap 000e[190] = ARI 1
     ecap 0019[1a0] = PCIe Sec 1 lane errors 0
    ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled 
                      0 VFs configured out of 0 supported
                      First VF RID Offset 0x0008, VF RID Stride 0x0004
                      VF Device ID 0x5803
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304 
     ecap 0017[200] = TPH Requester 1

# git status
On branch stable/13
Your branch is up to date with 'origin/stable/13'.

nothing to commit, working tree clean
13.1-PRERELEASE FreeBSD 13.1-PRERELEASE #1 stable/13-226bb05eb: Mon Feb 28 15:03:59 EST 2022 

Loader.conf is


t5fw_cfg_load="YES"
if_cxgbe_load="YES"

hw.cxgbe.pause_settings="0"
hw.cxgbe.attack_filter="1"
hw.cxgbe.drop_pkts_with_l3_errors="1"
hw.cxgbe.fcoecaps_allowed="0"

     ---Mike

Reply via email to