On 13/11/2017 22:41, Anish wrote: > Hi Andriy, > >>What's suspicious is that there are no interrupts either for the > ppt device on the host or the ahci device in the guest. > > I think most likely AMD IOMMU/amdvi is not enabled, by default it is disabled > and can be confirmed by sysctl. > > [root@ryzen /usr/home/anish]# sysctl hw.vmm.amdvi > hw.vmm.amdvi.domain_id: 3 > hw.vmm.amdvi.disable_io_fault: 0 > hw.vmm.amdvi.ptp_level: 4 > hw.vmm.amdvi.host_ptp: 1 > hw.vmm.amdvi.enable: 1 > hw.vmm.amdvi.count: 2 > > If hw.vmm.amdvi.enable is 0, set it and reload vmm and things should work.
I followed your how-to, so I didn't make this mistake :-) $ sysctl hw.vmm.amdvi hw.vmm.amdvi.domain_id: 3 hw.vmm.amdvi.disable_io_fault: 0 hw.vmm.amdvi.ptp_level: 4 hw.vmm.amdvi.host_ptp: 1 hw.vmm.amdvi.enable: 1 hw.vmm.amdvi.count: 1 Could it be that AMD-Vi support is broken in the hardware or BIOS? Are there any checks that I could do? Here is what I see in dmesg, but it is not a full dump of IVRS, of course: $ dmesg | egrep -i 'ivrs|ivh|amd-vi|mmu' Table 'IVRS' at 0xbdafaf80 ACPI: IVRS 0x00000000BDAFAF80 0000F8 (v01 AMD RD890S 00202031 AMD 00000000) AMD-Vi IVRS VAsize = 64 PAsize = 52 GVAsize = 0 flags:0 ivhd0: <AMD-Vi/IOMMU or ivhd> on acpi0 ivhd0: Unknown dev entry:0xff ivhd0: Flag:3e<PassPW,ResPassPW,Isoc,IotlbSup,Coherent> ivhd0: max supported paging level:7 restricting to: 4 ivhd0: device supported range [0x0 - 0xb01] ivhd0: device [0xa0 - 0xa0]config:d7<INIT,ExtInt,NMI,LINT0> ivhd0: PCI cap 0x10b540f@0x40 feature:1<IOTLB> pci0: <base peripheral, IOMMU> at device 0.2 (no driver attached) > On Mon, Nov 13, 2017 at 11:02 AM, Andriy Gapon <a...@freebsd.org > <mailto:a...@freebsd.org>> wrote: > > > I have a FreeBSD guest on a FreeBSD host, both amd64 head. > I decided to experiment with PCI pass-through and used an AHCI controller > for > the experiment. It seems like that partially works. The guest detects > the > controller and attaches a driver to it, the messages (reported > capabilities, > etc) look identical to those on the host. But the guest can not detect > any > disks behind the controller. From what I can tell, the guest has no > problems > accessing PCI configuration registers and a memory mapped BAR. Not sure > about > the I/O ports. What's suspicious is that there are no interrupts either > for the > ppt device on the host or the ahci device in the guest. Maybe that is > what > causes the failure to see the disks. > > When the guest driver attaches to the controller I see these messages on > the > host: > ppt0: attempting to allocate 1 MSI vectors (1 supported) > ppt0: using IRQ 265 for MSI > and in the guest: > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > > But vmstat -i does not report any interrupts. > > Just in case, the controller has some complex topology where the AHCI PCI > device > is behind two PCI-PCI bridges from the main bus. But I think that that > should > not matter and the guest needs to see only the AHCI device and the host > handles > the bridges. > > Here are some more messages from the guest: > ahci0: <HighPoint RocketRAID 640 AHCI SATA controller> port > 0x21c0-0x21c7,0x21c8-0x21cb,0x21d0-0x21d7,0x21d8-0x21db,0x21e0-0x21ef mem > 0xc000e000-0xc000efff irq 17 at device 9.0 on pci0 > ahci0: attempting to allocate 1 MSI vectors (1 supported) > ahci0: using IRQ 279 for MSI > ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS > ahci0: Caps: 64bit NCQ SNTF AL 6Gbps PM FBS PMD SSC PSC 32cmd eSATA 2ports > ahcich0: <AHCI channel> at channel 0 on ahci0 > ahcich0: Caps: CPD ESP FBSCP > ahcich1: <AHCI channel> at channel 1 on ahci0 > ahcich1: Caps: CPD ESP FBSCP > ahcich0: AHCI reset... > ahcich0: SATA connect time=100us status=00000123 > ahcich0: AHCI reset: device found > ahcich1: AHCI reset... > ahcich1: SATA connect time=100us status=00000123 > ahcich1: AHCI reset: device found > ahcich1: AHCI reset: device ready after 100ms > ahcich1: Poll timeout on slot 1 port 15 > ahcich1: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > ahcich0: AHCI reset: device ready after 200ms > ahcich0: Poll timeout on slot 1 port 15 > ahcich0: is 00000000 cs 00000002 ss 00000000 rs 00000002 tfd 150 serr > 00000000 > cmd 00318017 > And some more... > > Any ideas or suggestions? > Or maybe some additional information from me? > > Thank you! > > -- > Andriy Gapon > _______________________________________________ > freebsd-virtualization@freebsd.org > <mailto:freebsd-virtualization@freebsd.org> mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization > <https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization> > To unsubscribe, send any mail to > "freebsd-virtualization-unsubscr...@freebsd.org > <mailto:freebsd-virtualization-unsubscr...@freebsd.org>" > > -- Andriy Gapon _______________________________________________ freebsd-virtualization@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-virtualization To unsubscribe, send any mail to "freebsd-virtualization-unsubscr...@freebsd.org"