On Tue, Aug 24, 2021 at 08:31:09PM +0200, Mark Kettenis wrote:
| > | I don't see any trace of it. What server model is this? Is this by
| > | any chance a dual socket server with only one socket populated?
| >
| > Well, it's SuperMicro, so it's all a bit hazy. At any rate, the
| > motherboard is the M11SDV-8CT-LN4F if that helps. It's single socket
| > (only one CPU), however, the PCIe to NVMe adapter has two slots. I'll
| > move the SSD to the other slot to see if that makes a difference.
| >
| > Lo and behold:
| >
| > sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPELKX02, VCV1>
| > sd0: 1907729MB, 512 bytes/sector, 3907029168 sectors
| >
| > OK, that's unexpected. But it works :)
| >
| > Full dmesg and pcidump -v again included. If acpidump and pcidump
| > -vxxx are useful, I'll happily send those too, but since things now
| > work I think we can close the case without them.
|
| So:
|
| > pchb2 at pci0 dev 3 function 0 "AMD 17h PCIE" rev 0x00
| > ppb3 at pci0 dev 3 function 1 "AMD 17h PCIE" rev 0x00: msi
| > pci4 at ppb3 bus 5
| > nvme0 at pci4 dev 0 function 0 "Intel SSD DC" rev 0x00: msix, NVMe 1.2
| > nvme0: INTEL SSDPELKX020T8, firmware VCV10370, serial PHLJ021200RZ2P0J
| > scsibus1 at nvme0: 129 targets, initiator 0
| > sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPELKX02, VCV1>
| > sd0: 1907729MB, 512 bytes/sector, 3907029168 sectors
|
| This tells us that this NVMe adapter card does not have a PCIe switch
| on it. So the PCIe lanes must be passed straight through to the NVMe
| slots. If you look at the documentation its says in chapter 3:
Explaining for my understanding: ppb3 hangs straight of the CPU (it's
AMD 17h PCIE); then there's pci4 on top of that where the nvme0 device
is found. There's no extra ppb in between, so that tells you there's
no PCIe switch on the adapter card.
But I'm curious what documentation you consulted for this. Chapter 3
of the motherboard's manual doesn't mention anything about this.
| Depending on the system, motherboard, and BIOS version, the
| following BIOS settings may be necessary for the proper operation of
| M.2 NVMe drivers:
|
| * Having the CPU IOU settings set to x4x4x4x4 PCIe bifurcation. This
| option may be found under BIOS Setup -> Advanced -> Chipset
| Configuration -> North Bridge -> IIO Configuration -> CPU
| Configuration -> IOU Setting -> x4x4x4x4.
|
| So in order to support both M.2 slots on the card, the card relies on
| BIOS support for bifurcation. So you may need to tweak some settings
| there if you want to use the other M.2 slot on the card. However,
| your motherboard isn't listed as supported so that BIOS option may not
| exist.
Well, with this new found knowledge, I went back to the BIOS and found
"CPU SLOT7 PCI-E 3.0 X16 Bifurcation" which I could set to x4x4x4x4.
Since there's only one PCIe slot on the motherboard, I'm assuming this
option is about that slot.
And indeed, when I set that option to x4x4x4x4, the NVME SSD works
just fine in the other slot. I'm reinstalling now in UEFI mode, will
follow up with a new dmesg ocne it's installed, but the relevant bits
from bsd.rd dmesg are:
ppb3 at pci0 dev 3 function 3 "AMD 17h PCIE" rev 0x00: msi
pci4 at ppb3 bus 5
nvme0 at pci4 dev 0 function 0 "Intel SSD DC" rev 0x00: msix, NVMe 1.2
nvme0: INTEL SSDPELKX020T8, firmware VCV10370, serial PHLJ021200RZ2P0J
scsibus0 at nvme0: 129 targets, initiator 0
sd0 at scsibus0 targ 1 lun 0: <NVMe, INTEL SSDPELKX02, VCV1>
sd0: 1907729MB, 512 bytes/sector, 3907029168 sectors
So the function changed (from 1 to 3), but otherwise things are pretty
similar.
Again, thanks Mark! This whole bifurcation stuff is new to me, glad
to have learned about it.
For the archives: if your NVMe SSD doesn't work in a two-slot
PCIe-to-M.2 adapter, one option is to move the SSD to the other slot;
alternatively, try to find a PCIe-bifurcation option in your BIOS.
Cheers,
Paul
--
>++++++++[<++++++++++>-]<+++++++.>+++[<------>-]<.>+++[<+
+++++++++++>-]<.>++[<------------>-]<+.--------------.[-]
http://www.weirdnet.nl/
