Hello,
Here are my findings after 2 weeks of dtb hammering. ## CM3588 NAS / RK3588 PCIe bring-up findings on OpenBSD I’ve been debugging NVMe instability on a FriendlyElec CM3588 NAS running OpenBSD/arm64. The board has 4 M.2 NVMe sockets. The main symptom is that one or more NVMe devices enumerate, but access to some paths stalls the system, especially during early probing or when softraid is enabled. ### Hardware / controller mapping >From Linux DTS comments, board behavior, and pinctrl/reset mapping, the >socket-to-controller mapping appears to be: | Socket | Controller | Address | Notes | | ------ | -----------: | ---------- | ------------------ | | 1 | controller 0 | `fe150000` | consistently works | | 2 | controller 2 | `fe170000` | problematic | | 3 | controller 1 | `fe160000` | consistently works | | 4 | controller 3 | `fe180000` | most problematic | The reset pin mapping extracted from the DTB is: | Socket | Controller | pinctrl group | reset pin | | | ------ | ---------- | ------------- | --------- | - | | 1 | `fe150000` | `pcie3x4-rst` | GPIO4_B6 | | | 2 | `fe170000` | `pcie2-0-rst` | GPIO4_B4 | | | 3 | `fe160000` | `pcie3x2-rst` | GPIO4_B3 | | | 4 | `fe180000` | `pcie2-1-rst` | GPIO4_A2 | | ### Strong observed pattern The stable controllers are: * `fe150000` (socket 1) * `fe160000` (socket 3) The unstable controllers are: * `fe170000` (socket 2) * `fe180000` (socket 4) So the problem clusters around the “lane 1” paths rather than affecting all four sockets equally. ### Firmware observations With the original bootloader, U-Boot identified as NanoPC-T6, which was clearly wrong for this board. With Armbian/mainline-style U-Boot, the board identity became correct and all 4 NVMe devices could be seen by U-Boot: * `nvme scan` * `nvme info` showed all four Samsung drives. This means the board and routing are capable of bringing up all four links under firmware. ### OpenBSD behavior Under OpenBSD: * sockets 1 and 3 are reliably usable * socket 2 can sometimes be made to enumerate depending on DT changes * socket 4 (`fe180000`) is the main failure path Typical failure mode for socket 4: * device enumerates * NVMe attaches * then the system stalls when the device is actually accessed * boot usually only succeeds with `disable softraid` That suggests “link up” is not the real problem anymore; the problem is what happens after enumeration. ### DT findings 1. `rockchip,pipe-grf` matters Adding `rockchip,pipe-grf` to controller nodes changed behavior significantly and made some previously failing paths work. That suggests the GRF hookup is functionally required, not optional. 2. `data-lanes` changes behavior globally Changing the PCIe3 PHY `data-lanes` from the Linux/mainline-style split ordering to a sequential ordering changed which sockets came up: * one ordering gave only sockets 1 and 3 (data-lanes = <1 3 2 4> ) * another ordering gave sockets 1, 2, and 3 (data-lanes = <1 2 3 4> ) * socket 4 remained the main outlier ``` usb5 at ohci0: USB revision 1.0 uhub5 at usb5 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1 usb6 at ohci1: USB revision 1.0 uhub6 at usb6 configuration 1 interface 0 "Generic OHCI root hub" rev 1.00/1.00 addr 1 pci0 at dwpcie0 ppb0 at pci0 dev 0 function 0 "Rockchip RK3588" rev 0x00 pci1 at ppb0 bus 49 nvme0 at pci1 dev 0 function 0 "Samsung PM9C1a" rev 0x00: msix, NVMe 2.0 nvme0: Samsung SSD 990 EVO Plus 4TB, firmware 2B2QKXG7, serial S7U9NU0YA03521K < nvme socket 4 scsibus0 at nvme0: 2 targets, initiator 0 sd0 at scsibus0 targ 1 lun 0: <NVMe, Samsung SSD 990, 2B2Q> sd0: 3815447MB, 512 bytes/sector, 7814037168 sectors pci2 at dwpcie1 ppb1 at pci2 dev 0 function 0 "Rockchip RK3588" rev 0x00 pci3 at ppb1 bus 65 rge0 at pci3 dev 0 function 0 "Realtek RTL8125" rev 0x05: msix: RTL8125B, address 00:00:00:00:00:00 pci4 at dwpcie2 ppb2 at pci4 dev 0 function 0 "Rockchip RK3588" rev 0x00 pci5 at ppb2 bus 1 nvme1 at pci5 dev 0 function 0 "Samsung PM9C1a" rev 0x00: msix, NVMe 2.0 nvme1: Samsung SSD 990 EVO Plus 4TB, firmware 2B2QKXG7, serial S7U9NU0YA09134R <- nvme socket 1 scsibus1 at nvme1: 2 targets, initiator 0 sd1 at scsibus1 targ 1 lun 0: <NVMe, Samsung SSD 990, 2B2Q> sd1: 3815447MB, 512 bytes/sector, 7814037168 sectors pci6 at dwpcie3 ppb3 at pci6 dev 0 function 0 "Rockchip RK3588" rev 0x00 pci7 at ppb3 bus 17 nvme2 at pci7 dev 0 function 0 "Samsung PM9C1a" rev 0x00: msix, NVMe 2.0 nvme2: Samsung SSD 990 EVO Plus 4TB, firmware 2B2QKXG7, serial S7U9NU0YA09309A <- nvme socket 3 scsibus2 at nvme2: 2 targets, initiator 0 sd2 at scsibus2 targ 1 lun 0: <NVMe, Samsung SSD 990, 2B2Q> sd2: 3815447MB, 512 bytes/sector, 7814037168 sectors pci8 at dwpcie4 ppb4 at pci8 dev 0 function 0 "Rockchip RK3588" rev 0x00 pci9 at ppb4 bus 33 nvme3 at pci9 dev 0 function 0 "Samsung PM9C1a" rev 0x00: msix, NVMe 2.0 nvme3: Samsung SSD 990 EVO Plus 4TB, firmware 2B2QKXG7, serial S7U9NU0YA03531D <- nvme socket 2 scsibus3 at nvme3: 2 targets, initiator 0 sd3 at scsibus3 targ 1 lun 0: <NVMe, Samsung SSD 990, 2B2Q> sd3: 3815447MB, 512 bytes/sector, 7814037168 sectors scsibus4 at sdmmc0: 2 targets, initiator 0 sd4 at scsibus4 targ 1 lun 0: <Apacer, SD8GB, 0030> removable sd4: 7384MB, 512 bytes/sector, 15122432 sectors scsibus5 at sdmmc1: 2 targets, initiator 0 sd5 at scsibus5 targ 1 lun 0: <SD/MMC, A3A561, 0000> sd5: 59000MB, 512 bytes/sector, 120832000 sectors vscsi0 at root scsibus6 at vscsi0: 256 targets ``` This suggests the `data-lanes` property is affecting the global PHY split/routing behavior, but OpenBSD is not "realizing" the intended RK3588 topology the same way Linux does, this might be the reason behind the diff results by re-ordering the values in the data-lanes. 3. OpenBSD DT pinctrl skeleton exists but some content is missing (Not sure the true effect of this) The corresponding PCIe pinctrl structure exists in the OpenBSD DTB, but parts of it appear empty, while vendor/Linux-style DTBs populate the pinctrl groups with actual pin definitions. That may contribute to incomplete sideband/reset setup. (?! unsure) ### Linux upstream findings that appear directly relevant Two Linux changes look highly relevant to this board: 1. CM3588 NAS regulators: Linux added a fix titled “Enable all PCIe regulators on CM3588 NAS” because some SSDs could still enumerate even with regulators not fully enabled, but then would hang on access. That seems matche the OpenBSD symptom very closely. 2. RK3588 PCIe bifurcation: Linux also fixed RK3588 PCIe3 PHY bifurcation programming, specifically the GRF programming behind the lane split handling. CM3588 NAS was explicitly called out as one of the boards affected by that issue. (https://github.com/torvalds/linux/commit/f8020dfb311d2b6cf657668792aaa5fa8863a7dd) ### Current interpretation At this point, my working theory is that the issue might not basic enumeration, but incomplete RK3588-specific runtime support for the problematic lane/controller paths, especially `fe180000`. The evidence now suggests: * firmware can bring up all 4 links * U-Boot can enumerate all 4 NVMe devices * OpenBSD still fails when using the socket-4 / `fe180000` path * regulator enable state and RK3588 PCIe3/GRF lane handling are likely involved * DT-only changes improve behavior, but do not fully solve it, its able to boot with softraid disabled and even get the sector and basic device info but after the boot is complet the device is "Non-functional" a simple disklable sd0 hangs.. ### Minimal reproducible summary * Board: FriendlyElec CM3588 NAS * SoC: RK3588 * 4 identical Samsung 990 EVO Plus 4TB NVMe drives * U-Boot: Armbian/mainline-style U-Boot with correct CM3588 NAS identity * Result: U-Boot sees all 4 NVMe devices * OpenBSD result: * sockets 1 and 3 usable * socket 2 variable depending on DT changes * socket 4 (`fe180000`) enumerates then destabilizes access / boot * system often only boots with `disable softraid` NOTE: I tested this with a decompliled version from openbsd ports and made a few adjustments, if i can provide it if any one is interested. ### Info request Any one has any sugestions / information that were i could understand the following: * RK3588 PCIe3 PHY lane split / bifurcation handling * GRF programming for lane-1 paths * regulator handling for CM3588 NAS M.2 slots * controller/runtime handling for `fe170000` / `fe180000` after link-up, not just at attach time. Best Regards Tiago On Saturday, 17 January 2026 at 01:53, Dave Polaschek <[email protected]> wrote: > On Fri, Jan 16, 2026, at 18:27, Mike Larkin wrote: > > On Fri, Jan 16, 2026 at 05:27:27PM -0700, Dave Polaschek wrote: > >> Thanks for sharing this info. I’m supposed to be receiving my RK3588 > >> upgrade board for my MNT Pocket Reform in early February, and using this, > >> maybe I’ll get it running OpenBSD sooner rather than later. Hope so! > > > > Not sure if the panel will work... IIRC patrick@ had to do some special > > things > > to get the panel working on the existing iMX machines.. > > Yeah, it's a portrait panel being used landscape, if nothing else. I was > planning to try and install 7.8 on the iMX version I have now, but after > having to send it to Germany for warranty repairs, and then fiddle with the > backplate to get WiFi that could stay connected long enough to do an "apt > update," I figure I'll just wait and see what I can find when the RK3588 > arrives. > > But I figure with this latest info, I've got at least one fewer roadblock in > the way, and I'd rather have it as an OpenBSD machine than Debian for a few > reasons, and it's a lot easier to switch OSes before I start using it as a > daily driver. > > >> -DaveP > >> > >> On Fri, Jan 16, 2026, at 17:12, 0x4E0x650x6F wrote: > >> > Hello OpenBSD arm64 team, > >> > > >> > I wanted to share some notes from a successful OpenBSD/arm64 > >> > bring-up on a FriendlyElec CM3588 Plus (NanoPC-T6, RK3588, 32 GB LPDDR5). > >> > This took a bit of digging, and I think one detail in particular may > >> > be useful to others working with Rockchip boards. > >> > > >> > Summary > >> > OpenBSD 7.8 boots reliably from eMMC via U-Boot + EFI once disk layout > >> > and > >> > bootloader offsets are handled correctly. > >> > The system now boots cleanly with no SD card and no manual U-Boot > >> > interaction. > >> > > >> > Key finding (important) > >> > On RK3588, U-Boot must be located at sector 64 (32 KiB offset). > >> > If the EFI/MSDOS partition or GPT header is created too early on the > >> > disk, > >> > it overwrites the expected U-Boot location. > >> > > >> > In that case: > >> > > >> > SPL runs > >> > U-Boot is not found > >> > The system falls back to recovery / mask-like behaviour > >> > > >> > The fix was: > >> > > >> > 1. Ensure the first partition starts *after* sector 64 > >> > 2. Explicitly write U-Boot with: > >> > > >> > ``` > >> > dd if=u-boot-rockchip.bin of=/dev/rsdXc bs=512 seek=64 conv=fsync > >> > ``` > >> > > >> > Once this is done, the boot chain works reliably: > >> > ROM → SPL → U-Boot → EFI → OpenBSD BOOTAA64.EFI → kernel. > >> > > >> > DTB > >> > The DTB shipped in the OpenBSD install sets: > >> > > >> > ``` > >> > rk3588-friendlyelec-cm3588-nas.dtb > >> > ``` > >> > > >> > is the correct match for the CM3588 Plus. > >> > Vendor DTS files differ between CM3588 and CM3588-Plus (rev06 vs rev09), > >> > and using the wrong one causes subtle issues (USB, clocks, PHYs). > >> > > >> > eMMC interference: > >> > I also ran into a hang around USB init (“ohci0 at mainbus0”). > >> > This turned out not to be a USB driver issue, but leftover vendor > >> > data on eMMC interfering with boot. > >> > Zeroing the first 1 GiB of eMMC resolved it: > >> > > >> > ``` > >> > dd if=/dev/zero of=/dev/mmcblk2 bs=1M count=1024 > >> > ``` > >> > > >> > After that, the installer proceeded normally. > >> > > >> > I’m happy to provide more detailed notes, disk layouts, > >> > or testing if useful. > >> > Thanks for the excellent arm64 support once the boot > >> > geometry was understood, everything fell into place nicely. > >> > > >> > MY "hat" is officially off for the amazing work on this OS.... > >> > > >> > PS: > >> > The dmesg was already sent I will try to tune some things that > >> > I still haven't tested yet specifically the nvme sockets. > >> > > >> > Thanks Mike Larkin again for your input. > >> > > >> > Best regards, > >> > Tiago > >> > > >> > > >> > > >> > Sent with Proton Mail secure email. > >> > > >> > On Thursday, 15 January 2026 at 12:15, 0x4E0x650x6F > >> > <[email protected]> wrote: > >> > > >> >> Hi all, > >> >> > >> >> Thanks for the response here's where I'm at. > >> >> > >> >> Summary: > >> >> CM3588-Plus (RK3588): OpenBSD 7.8 RAMDISK boots via u-boot-rk3588/EFI, > >> >> stalls after ohci0 attach; vendor DT differs for Plus. > >> >> > >> >> Details: > >> >> > >> >> I’m testing OpenBSD/arm64 on a FriendlyElec CM3588-Plus (RK3588, 32GB). > >> >> I can boot the OpenBSD 7.8 RAMDISK kernel via SD card using OpenBSD’s > >> >> u-boot-rk3588 package and EFI (BOOTAA64.EFI), but the kernel > >> >> consistently stalls during early device attach. > >> >> > >> >> Hardware > >> >> Board: FriendlyElec CM3588-Plus > >> >> SoC: RK3588 > >> >> RAM: 32GB > >> >> Serial console: UART2 (serial@feb50000), 1,500,000 8N1 (DT stdout-path > >> >> uses serial2:1500000n8) > >> >> Boot path / versions > >> >> SD card: miniroot78.img (dd) > >> >> U-Boot: u-boot-rk3588 (U-Boot 2025.07; boots via bootflow scan / > >> >> efi_mgr) > >> >> EFI loader: OpenBSD/arm64 BOOTAA64 1.22 > >> >> Kernel: OpenBSD 7.8 (RAMDISK) #38 ... > >> >> > >> >> Boot log excerpt shows normal early init (PSCI/GIC/EFI, memory, SCMI, > >> >> clock, GPIO, etc.) then USB host attach, then stall. > >> >> > >> >> Last lines printed: > >> >> xhci0 at mainbus0, xHCI 1.10 > >> >> usb0 at xhci0: USB revision 3.0 > >> >> uhub0 at usb0 configuration 1 interface 0 "Generic xHCI root hub" ... > >> >> ehci0 at mainbus0 > >> >> usb1 at ehci0: USB revision 2.0 > >> >> uhub1 at usb1 configuration 1 interface 0 "Generic EHCI root hub" ... > >> >> ohci0 at mainbus0 > >> >> > >> >> After this: > >> >> no further output on serial > >> >> no panic/backtrace > >> >> no watchdog reset > >> >> > >> >> UKC attempt > >> >> I booted with UKC (boot -c) and issued: > >> >> disable rkusbphy0 > >> >> disable rkusbphy1 > >> >> disable rkusbphy2 > >> >> disable rkusbphy3 > >> >> disable dwc3 > >> >> disable usbdrd > >> >> quit > >> >> > >> >> UKC produced no confirmation output, and the same devices still > >> >> attached (rkusbphy0-3, xhci0/ehci0/ohci0) and stalled at the same point. > >> >> > >> >> DTB / board variant note || doubts > >> >> > >> >> OpenBSD provides rk3588-friendlyelec-cm3588-nas.dtb (dtb-6.14 / 7.8). > >> >> However FriendlyElec’s vendor kernel uses different DT sources for > >> >> CM3588 vs CM3588-Plus: > >> >> CM3588: rk3588-nanopi6-rev09.dts > >> >> CM3588-Plus: rk3588-nanopi6-rev06.dts > >> >> > >> >> Diffing OpenBSD cm3588-nas DT against the vendor CM3588-Plus DT shows > >> >> substantial USB-related differences (DWC3 node structure/compatibles, > >> >> clocks/clock-names, power-domain phandles, explicit reset lines, > >> >> EHCI/OHCI companion+PHY wiring), so the shipped DTB may not match > >> >> CM3588-Plus USB/PHY/power wiring. (????) > >> >> > >> >> I can provide full serial logs and the relevant DT diff hunks if needed. > >> >> (after a very long or deal not sure if I bricked the device or not last > >> >> night but I plan on keep hitting this thing until (hopefully until it > >> >> runs). > >> >> > >> >> > >> >> Best Regards > >> >> Tiago. > >> >> > >> >> > >> >> Sent with Proton Mail secure email. > >> >> > >> >> > >> >> On Thursday, 8 January 2026 at 06:09, Mike Larkin [email protected] > >> >> wrote: > >> >> > >> >> > On Wed, Jan 07, 2026 at 05:02:24PM +0000, 0x4E0x650x6F wrote: > >> >> > > >> >> > > Hi All, > >> >> > > > >> >> > > I finally decided to migrate to OpenBSD recently, > >> >> > > and I was thinking if i could also use openbsd > >> >> > > on CM3588 Plus (its a NAS board), uses rockchip > >> >> > > rk3588. > >> >> > > I noticed that a device with the same chip is > >> >> > > supported I trying to find any documentation / help > >> >> > > that could give me some guidance. > >> >> > > > >> >> > > I would be willing to assist to the best of my abilities, > >> >> > > I have a device that is not in active use that, > >> >> > > i could use as a test platform. > >> >> > > > >> >> > > Best Regards > >> >> > > Tiago Carvalho. > >> >> > > >> >> > Looks like a generic cm3588 board, you could try the rock5a DTB or > >> >> > something > >> >> > similar. probably would get you reasonably far then tweak. > >> >> > > >> >> > unless you already have the DTB, say, from linux... > >> >> > > >> >> > -ml > >
