> Date: Tue, 24 Aug 2021 19:18:08 +0200
> From: Paul de Weerd <we...@weirdnet.nl>
>
> Hi Mark,
>
> On Tue, Aug 24, 2021 at 06:15:50PM +0200, Mark Kettenis wrote:
> | > Perhaps it's because it lives behind a riser card, on a PCIe to NVMe
> | > adapter (SuperMicro's AOC-SLG3-2M2) - this because the motherboard
> | > only takes M.2 SSD's upto 80mm, but the Intel SSD I have is 110mm
> | > (Intel SSD DC P4511).
> | >
> | > Any idea if this is fixable? Included are full dmesg (with ipmi
> | > enabled), pcidump -v and (compressed) pcidump -vxxx output and a
> | > tarball with the contents of /var/db/acpi.
> |
> | I don't see any trace of it. What server model is this? Is this by
> | any chance a dual socket server with only one socket populated?
>
> Well, it's SuperMicro, so it's all a bit hazy. At any rate, the
> motherboard is the M11SDV-8CT-LN4F if that helps. It's single socket
> (only one CPU), however, the PCIe to NVMe adapter has two slots. I'll
> move the SSD to the other slot to see if that makes a difference.
>
> Lo and behold:
>
> sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPELKX02, VCV1>
> sd0: 1907729MB, 512 bytes/sector, 3907029168 sectors
>
> OK, that's unexpected. But it works :)
>
> Full dmesg and pcidump -v again included. If acpidump and pcidump
> -vxxx are useful, I'll happily send those too, but since things now
> work I think we can close the case without them.
So:
> pchb2 at pci0 dev 3 function 0 "AMD 17h PCIE" rev 0x00
> ppb3 at pci0 dev 3 function 1 "AMD 17h PCIE" rev 0x00: msi
> pci4 at ppb3 bus 5
> nvme0 at pci4 dev 0 function 0 "Intel SSD DC" rev 0x00: msix, NVMe 1.2
> nvme0: INTEL SSDPELKX020T8, firmware VCV10370, serial PHLJ021200RZ2P0J
> scsibus1 at nvme0: 129 targets, initiator 0
> sd0 at scsibus1 targ 1 lun 0: <NVMe, INTEL SSDPELKX02, VCV1>
> sd0: 1907729MB, 512 bytes/sector, 3907029168 sectors
This tells us that this NVMe adapter card does not have a PCIe switch
on it. So the PCIe lanes must be passed straight through to the NVMe
slots. If you look at the documentation its says in chapter 3:
Depending on the system, motherboard, and BIOS version, the
following BIOS settings may be necessary for the proper operation of
M.2 NVMe drivers:
* Having the CPU IOU settings set to x4x4x4x4 PCIe bifurcation. This
option may be found under BIOS Setup -> Advanced -> Chipset
Configuration -> North Bridge -> IIO Configuration -> CPU
Configuration -> IOU Setting -> x4x4x4x4.
So in order to support both M.2 slots on the card, the card relies on
BIOS support for bifurcation. So you may need to tweak some settings
there if you want to use the other M.2 slot on the card. However,
your motherboard isn't listed as supported so that BIOS option may not
exist.
Cheers,
Mark
