Hi Navdeep,
I updated a pair of releng13 boxes today as I saw there are new
drivers/firmware. However, the firmware seems to fail on both in
different ways
t5nex0: <Chelsio T540-CR> mem
0x91300000-0x9137ffff,0x90000000-0x90ffffff,0x91984000-0x91985fff irq 17
at device 0.4 on pci2
t5nex0: failed to load firmware module: 2. cfg 0, fw 0
t5nex0: firmware on card (1.25.0.40) is older than the version bundled
with this driver, but the driver cannot take corrective action because
it is unable to load the firmware module.
t5nex0: KLD with default config is not available.
t5nex0: failed to upload config file to card: 2.
t5nex0: failed (2) to configure card with "default" profile, will fall
back to a basic configuration and retry.
cxl0: <port 0> on t5nex0
cxl0: Ethernet address: 00:07:43:29:98:90
cxl0: 12 txq, 8 rxq (NIC)
cxl1: <port 1> on t5nex0
cxl1: Ethernet address: 00:07:43:29:98:98
cxl1: 12 txq, 8 rxq (NIC)
cxl2: <port 2> on t5nex0
cxl2: Ethernet address: 00:07:43:29:98:a0
cxl2: 12 txq, 8 rxq (NIC)
cxl3: <port 3> on t5nex0
cxl3: Ethernet address: 00:07:43:29:98:a8
cxl3: 12 txq, 8 rxq (NIC)
t5nex0: PCIe gen3 x8, 4 ports, 34 MSI-X interrupts, 84 eq, 33 iq
And on another adapter, different box
t5nex0: <Chelsio T520-LL-CR> mem
0xb2300000-0xb237ffff,0xb1000000-0xb1ffffff,0xb28a0000-0xb28a1fff irq 16
at device 0.4 on pci1
t5nex0: command 0x4 in mbox 4 timed out (0x80000000).
t5nex0: mbox 4 cmdsent 04a0000000000001 0f41000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
t5nex0: mbox 4 current 04a0000000000001 0f41000000000000
0000000000000000 0000000000000000 0000000000000000 0000000000000000
0000000000000000 0000000000000000
t5nex0: firmware reports adapter error: Crash (0x80000000)
t5nex0: device log follows.
Seq# Tstamp Level Facility Message
0 507376 INFO CORE log initialized @
0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000
1 510262 INFO CORE bootstrap firmware
took 605 msecs to run
2 511784 INFO CORE Serial Configuration
version: 0x11101000 VPD version: 0x2
3 511785 NOTICE CORE pcie: Serial
Configuration SPARE3 has Cookie Generation enabled.
4 511786 INFO CORE pcie: npf 7 (pfbitmap
0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4
t5nex0: encountered fatal error, adapter stopped.
t5nex0: firmware reports adapter error: Crash (0x80000000)
t5nex0: device log follows.
Seq# Tstamp Level Facility Message
0 507376 INFO CORE log initialized @
0x200a0000 size 32768 (128 entries) fwrev 0x011a0200 pcie_fw 0x00000000
1 510262 INFO CORE bootstrap firmware
took 605 msecs to run
2 511784 INFO CORE Serial Configuration
version: 0x11101000 VPD version: 0x2
3 511785 NOTICE CORE pcie: Serial
Configuration SPARE3 has Cookie Generation enabled.
4 511786 INFO CORE pcie: npf 7 (pfbitmap
0x7f) nvf 64 (pf 0..7 0x1010101000000000) vfstride 4
t5nex0: failed to connect to the firmware: 6, 0. PCIE_FW 0x80000000
t5nex0: error during attach, adapter is now in recovery mode.
none0@pci0:1:0:5: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5511 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T520-LL-CR Unified Wire Storage Controller'
class = mass storage
subclass = SCSI
bar [10] = type Memory, range 64, base 0xb2280000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0xb2200000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0xb2890000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
max read 512
link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)
cap 11[b0] = MSI-X supports 40 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5811
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
none1@pci0:1:0:6: class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5611 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T520-LL-CR Unified Wire Storage Controller'
class = serial bus
subclass = Fibre Channel
bar [10] = type Memory, range 64, base 0xb2100000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0xb2080000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0xb2880000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
max read 512
link x8(x8) speed 8.0(8.0) ASPM disabled(L0s/L1)
cap 11[b0] = MSI-X supports 40 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 1 fatal 0 non-fatal 0 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5811
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
t5iov3@pci0:2:0:3: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5003 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T540-CR Unified Wire Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0x91480000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0x91400000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0x91986000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 8 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
max read 512
link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
cap 11[b0] = MSI-X supports 8 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 16 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5803
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
t5nex0@pci0:2:0:4: class=0x020000 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5403 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T540-CR Unified Wire Ethernet Controller'
class = network
subclass = ethernet
bar [10] = type Memory, range 64, base 0x91300000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0x90000000, size 16777216,
enabled
bar [20] = type Memory, range 64, base 0x91984000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR
max read 4096
link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
cap 11[b0] = MSI-X supports 128 messages, enabled
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5803
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
none7@pci0:2:0:5: class=0x010000 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5503 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T540-CR Unified Wire Storage Controller'
class = mass storage
subclass = SCSI
bar [10] = type Memory, range 64, base 0x91280000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0x91200000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0x91982000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
max read 512
link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
cap 11[b0] = MSI-X supports 40 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5803
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
none8@pci0:2:0:6: class=0x0c0400 rev=0x00 hdr=0x00 vendor=0x1425
device=0x5603 subvendor=0x1425 subdevice=0x0000
vendor = 'Chelsio Communications Inc'
device = 'T540-CR Unified Wire Storage Controller'
class = serial bus
subclass = Fibre Channel
bar [10] = type Memory, range 64, base 0x91100000, size 524288,
enabled
bar [18] = type Memory, range 64, base 0x91080000, size 524288,
enabled
bar [20] = type Memory, range 64, base 0x91980000, size 8192, enabled
cap 01[40] = powerspec 3 supports D0 D3 current D0
cap 05[50] = MSI supports 32 messages, 64 bit, vector masks
cap 10[70] = PCI-Express 2 endpoint max data 256(2048) FLR RO
max read 512
link x8(x8) speed 8.0(8.0) ASPM L0s/L1(L0s/L1)
cap 11[b0] = MSI-X supports 40 messages
Table in map 0x20[0x0], PBA in map 0x20[0x1000]
cap 03[d0] = VPD
ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
ecap 0003[170] = Serial 1 0000000000000000
ecap 000e[190] = ARI 1
ecap 0019[1a0] = PCIe Sec 1 lane errors 0
ecap 0010[1c0] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI
disabled
0 VFs configured out of 0 supported
First VF RID Offset 0x0008, VF RID Stride 0x0004
VF Device ID 0x5803
Page Sizes: 4096 (enabled), 8192, 65536, 262144,
1048576, 4194304
ecap 0017[200] = TPH Requester 1
# git status
On branch stable/13
Your branch is up to date with 'origin/stable/13'.
nothing to commit, working tree clean
13.1-PRERELEASE FreeBSD 13.1-PRERELEASE #1 stable/13-226bb05eb: Mon Feb
28 15:03:59 EST 2022
Loader.conf is
t5fw_cfg_load="YES"
if_cxgbe_load="YES"
hw.cxgbe.pause_settings="0"
hw.cxgbe.attack_filter="1"
hw.cxgbe.drop_pkts_with_l3_errors="1"
hw.cxgbe.fcoecaps_allowed="0"
---Mike
