Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Paige Thompson]

Hey guys, 

As you probably know there is a workaround to use i440fx instead of q35 to get 
the R9 390 working with VFIO. However, I have managed to get it working with 
Q35 by adding PCIe upstream and downstream ports to the configuration. You've 
probably noticed that the R9 390 card works fine with Q35 up until the point in 
which you install the AMD drivers and then it starts crashing. As near as I can 
tell the driver simply doesn't like it if you attach the video card directly to 
the root complex. 

Here is a link to the configuration that I am using: 
https://gitlab.com/snippets/1788426#note_127191397 
For the entire work log :  https://gitlab.com/snippets/1788426

I'd also like to point out that this also works with a Linux guest as I 
confirmed a moment ago. I never could get Linux to work with either i440fx or 
q35 before I tried using a pcie bridge in q35. My understanding of why Windows 
would work with i440fx is unclear but I know that pcie devices essentially 
speak the same as pci devices and there are no pcie devices on i440fx 
platforms; still whatever the amd drivers check for is somehow circumvented in 
that particular configuration. 

I'm surprised nobody knew to suggest this, I sort of figured this out based on 
something I read somewhere about how it's a bad idea to allow devices to talk 
directly as discrete devices to the root complex anyway. So, I gave this a try 
and low and behold it worked. It won't give you any performance improvements, 
but q35 is a more accurate emulation of a modern system and may help with other 
aspects of troubleshooting, it just requires more topology complexity to be 
configured correctly.

Hope that helps, 

Paige Thompson 
erra...@yourstruly.sx

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Philip Soares]

On 3 Apr 2017, at 13:21, Graham Neville wrote:


Cheers for that. Added it to my config and set hugespaces via kernel
command line.

BOOT_IMAGE=/vmlinuz-linux 
root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw

quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
isolcpus=0-9


Are those the core and thread siblings on that CPU?

For a 10 core CPU I reserve a few for VMs like:

nohz_full=2-9,12-19 rcu_nocbs=2-9,12-19 isolcpus=2-7,12-17


kvm_amd.avic=1 hugepages=12188


Does the CPU support 1G pages? With 1G pages I reserve 32G for VMs like:

default_hugepagesz=1G hugepagesz=1G hugepages=32 
transparent_hugepage=never



I also set iothreads, but can't seem to change my disks from sata to
virtio. Whenever I try I get a windows bluescreen on boot.

I'm not seeing any interrupts for when the GPU is in use:


I don't have a Ryzen CPU so not sure about this.

I think someone just said yesterday on this list, in response to my 
question about it, that they had them with Ryzen, if I read it right.



 PIN:  0  0  0  0  0
0  0  0  0  0  0
0  0  0  0  0   Posted-interrupt
notification event
 PIW:  0  0  0  0  0
0  0  0  0  0  0
0  0  0  0  0   Posted-interrupt 
wakeup

event


Here's my current XML:




  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  16384000
  16384000
  

  
  8
  4
  












  
  
hvm
/home/virtualguests/windows10/ovmf_code_x64.bin
/home/virtualguests/windows10/ovmf_vars_x64.bin

  
  




  
  
  


  


  
  

  
  




  
  destroy
  restart
  restart
  


  
  
/usr/bin/qemu-system-x86_64

  
  file='/home/virtualguests/windows10/windows10-c-nas.qcow2'/>

  
  unit='0'/>



  
  
  
  unit='1'/>



  


  
  


  
  


  
  


  



  
  


  
  
  


  
  
  


  


  


  


  




  
  
  
  
  
  
  

   
  


  

   
  


  


  


  


  


  


  


  


  


  


  


  


  


  
function='0x0'/>

  
  
  


  
function='0x1'/>

  
  


  


On Mon, Apr 3, 2017 at 5:13 PM,  wrote:




Might perform better.

Also, yes Hugepages might be helpful too.

Also, would you mind observing whether interrupts are posted when the 
GPU

is in use in pass-through?

watch -d cat /proc/interrupts

Look at the bottom for PIN (Posted-interrupt notification event) and 
PIW

(Posted-interrupt wakeup event).

Thanks!

On 30 Mar 2017, at 15:46, Graham Neville wrote:

Finally gotten to the bottom of this, this the help of your XML file 
I'm
able to run with a Q35 setup, I'm also able to just passthrough the 
CPU

features. Thanks.

I discovered the issue with the guest crashing was actually due to my
graphics card overheating! Noticed that it was hitting 99C and then 
guest

would crash.

The setup is pretty sweet now, I just want to change my drives from 
sata
to virtio which I believe will be quicker. Then also investigate 
HugePages


Here's my Kernel params:

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux 
root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26
rw quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 
video=efifb:off

isolcpus=0-7

Here's my final XML file:


  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  8388608
  8388608
  4
  




  
  
hvm
/home/
virtualguests/windows10/ovmf_code_x64.bin
/home/virtualguests/windows10/ovmf_vars_x64.bin

  
  




  
  
  


  

  
  
  




  
  destroy
  restart
  restart
  


  
  
/usr/bin/qemu-system-x86_64

  
  file='/home/virtualguests/windows10/windows10-c-nas.qcow2'/>

  
  unit='0'/>



  
  
  
  unit='1'/>



  


  
  


  
  


  
  


  



  
  


  
  
  


  
  
  


  


  


  


  




  
  
  
  
  
  

   
  


  

   
  


  


  
  

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[globalgorrilla]

On 4 Apr 2017, at 20:37, Daimon Wang wrote:

Hi Graham,    Windows crash after switch to virtio-disk because its 
boot loader doesn't have driver for the disk.


^ this, and ...

    Reinstall Windows with virtio-disk would fix the issue (you'll 
need the virtio-disk driver during installation).I'm not sure 
if there's any way to install the virtio-disk driver to an existing 
windows.


To avoid that you might add a second empty, or temporary, virtio disk to 
the VM.


Then either add the vfio drivers ISO to the VM as well or download and 
mount it in the VM.


Once the driver is installed you can shutdown and change the C:\ disk to 
virtio.




Regards,Daimon

On Tuesday, April 4, 2017 4:21 AM, Graham Neville 
 wrote:



 Cheers for that. Added it to my config and set hugespaces via kernel 
command line.
BOOT_IMAGE=/vmlinuz-linux 
root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw quiet amd_iommu=on 
vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off isolcpus=0-9 
kvm_amd.avic=1 hugepages=12188
I also set iothreads, but can't seem to change my disks from sata to 
virtio. Whenever I try I get a windows bluescreen on boot.


I'm not seeing any interrupts for when the GPU is in use:
 PIN:  0  0  
0  0  0  
0  0  0  
0  0  0  
0  0  0  
0  0   Posted-interrupt notification event
 PIW:  0  0  
0  0  0  
0  0  0  
0  0  0  
0  0  0  
0  0   Posted-interrupt wakeup event


Here's my current XML:



  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  16384000
  16384000
  
    
  
  8
  4
  
    
    
    
    
    
    
    
    
    
    
    
    
  
  
    hvm
    type='pflash'>/home/virtualguests/windows10/ovmf_code_x64.bin

    /home/virtualguests/windows10/ovmf_vars_x64.bin
    
  
  
    
    
    
    
      
      
      
    
    
  
    
    
  
  
    
  
  
    
    
    
    
  
  destroy
  restart
  restart
  
    
    
  
  
    /usr/bin/qemu-system-x86_64
    
      
      file='/home/virtualguests/windows10/windows10-c-nas.qcow2'/>

      
  unit='0'/>

    
    
      
      
      
  unit='1'/>

    
    
      function='0x7'/>

    
    
      
      function='0x0' multifunction='on'/>

    
    
      
      function='0x1'/>

    
    
      
      function='0x2'/>

    
    
      function='0x2'/>

    
    
    
      
      function='0x0'/>

    
    
      
      
      function='0x0'/>

    
    
      
      
      function='0x0'/>

    
    
      function='0x0'/>

    
    
      
    
    
      
    
    
      
    
    
    
    
  
  
  
  
  
  
  function='0x0'/>

    
   
  
    
    
  
    
   
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
      function='0x0'/>

    
    
      
        function='0x0'/>

      
      file='/home/virtualguests/windows10/r9290.rom'/>
      function='0x0' multifunction='on'/>

    
    
      
        function='0x1'/>

      
      function='0x1'/>

    
    
  


On Mon, Apr 3, 2017 at 5:13 PM,  wrote:

 Might perform better. Also, yes Hugepages 
might be helpful too.Also, would you mind observing whether interrupts 
are posted when the GPU is in use in pass-through?watch -d cat 
/proc/interruptsLook at the bottom for PIN (Posted-interrupt 
notification event) and PIW (Posted-interrupt wakeup event).Thanks!On 
30 Mar 2017, at 15:46, Graham Neville wrote:
Finally gotten to the bottom of this, this the help of your XML file 
I'm able to run with a Q35 setup, I'm also able to just passthrough 
the CPU features. Thanks.
I discovered the issue with the guest crashing was actually due to my 
graphics card overheating! Noticed that it was hitting 99C and then 
guest would crash.
The setup is pretty sweet now, I just want to change my drives from 
sata to virtio which I believe will be quicker. Then also investigate 
HugePages


Here's my Kernel params:

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a- 
b92e-a4343ca20d26 rw quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002: 
aac8 video=efifb:off isolcpus=0-7


Here's my final XML file:

  Windows10
  d45c3b5f-be8a-41e8-a22c- 02e91c4c6245
  8388608
  8388608
  4
  
    
    
    
    
  
  
    hvm
    /home/ 
virtualguests/windows10/ovmf_ code_x64.bin

    /home/virtualguests/ windows10/ovmf_vars_x64.bin
    
  
  
    
    
    
    
      
      
      
    
    
  
    
  
  
  
    
    
    
    
  
  destroy
  restart
  restart
  
    
    
  
  
    /usr/bin/qemu- system-x86_64
    
      
      

 

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Daimon Wang]

Hi Graham,    Windows crash after switch to virtio-disk because its boot loader 
doesn't have driver for the disk.
    Reinstall Windows with virtio-disk would fix the issue (you'll need the 
virtio-disk driver during installation).I'm not sure if there's any way to 
install the virtio-disk driver to an existing windows.

Regards,Daimon 

On Tuesday, April 4, 2017 4:21 AM, Graham Neville  
wrote:
 

 Cheers for that. Added it to my config and set hugespaces via kernel command 
line.
BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw 
quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off 
isolcpus=0-9 kvm_amd.avic=1 hugepages=12188
I also set iothreads, but can't seem to change my disks from sata to virtio. 
Whenever I try I get a windows bluescreen on boot.

I'm not seeing any interrupts for when the GPU is in use: 
 PIN:  0  0  0  0  0  0 
 0  0  0  0  0  0  0  0 
 0  0   Posted-interrupt notification event
 PIW:  0  0  0  0  0  0 
 0  0  0  0  0  0  0  0 
 0  0   Posted-interrupt wakeup event

Here's my current XML:



  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  16384000
  16384000
  
    
  
  8
  4
  
    
    
    
    
    
    
    
    
    
    
    
    
  
  
    hvm
    /home/virtualguests/windows10/ovmf_code_x64.bin
    /home/virtualguests/windows10/ovmf_vars_x64.bin
    
  
  
    
    
    
    
      
      
      
    
    
  
    
    
  
  
    
  
  
    
    
    
    
  
  destroy
  restart
  restart
  
    
    
  
  
    /usr/bin/qemu-system-x86_64
    
      
      
      
  
    
    
      
      
      
  
    
    
      
    
    
      
      
    
    
      
      
    
    
      
      
    
    
      
    
    
    
      
      
    
    
      
      
      
    
    
      
      
      
    
    
      
    
    
      
    
    
      
    
    
      
    
    
    
    
  
  
  
  
  
  
  
    
   
  
    
    
  
    
   
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
      
    
    
      
        
      
      
      
    
    
      
        
      
      
    
    
  


On Mon, Apr 3, 2017 at 5:13 PM,  wrote:

 Might perform better. Also, yes Hugepages might be 
helpful too.Also, would you mind observing whether interrupts are posted when 
the GPU is in use in pass-through?watch -d cat /proc/interruptsLook at the 
bottom for PIN (Posted-interrupt notification event) and PIW (Posted-interrupt 
wakeup event).Thanks!On 30 Mar 2017, at 15:46, Graham Neville wrote:
Finally gotten to the bottom of this, this the help of your XML file I'm able 
to run with a Q35 setup, I'm also able to just passthrough the CPU features. 
Thanks.
I discovered the issue with the guest crashing was actually due to my graphics 
card overheating! Noticed that it was hitting 99C and then guest would crash.
The setup is pretty sweet now, I just want to change my drives from sata to 
virtio which I believe will be quicker. Then also investigate HugePages

Here's my Kernel params:

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a- b92e-a4343ca20d26 rw 
quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002: aac8 video=efifb:off 
isolcpus=0-7

Here's my final XML file:

  Windows10
  d45c3b5f-be8a-41e8-a22c- 02e91c4c6245
  8388608
  8388608
  4
  
    
    
    
    
  
  
    hvm
    /home/ virtualguests/windows10/ovmf_ 
code_x64.bin
    /home/virtualguests/ windows10/ovmf_vars_x64.bin
    
  
  
    
    
    
    
      
      
      
    
    
  
    
  
  
  
    
    
    
    
  
  destroy
  restart
  restart
  
    
    
  
  
    /usr/bin/qemu- system-x86_64
    
      
      
      
      
    
    
      
      
      
      
    
    
      
    
    
      
      
    
    
      
      
    
    
      
      
    
    
      
    
    
    
      
      
    
    
      
      
      
    
    
      
      
      
    
    
      
    
    
      
    
    
      
    
    
      
    
    
    
    
  
  
  
  
  
  
    
   
  
    
    
  
    
   
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
  
    
    
      
    
    
      
        
      
      
      
    
    
      
        
      
      
    
    
      
    
  




On Thu, Mar 30, 2017 

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Graham Neville]

Cheers for that. Added it to my config and set hugespaces via kernel
command line.

BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw
quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
isolcpus=0-9 kvm_amd.avic=1 hugepages=12188

I also set iothreads, but can't seem to change my disks from sata to
virtio. Whenever I try I get a windows bluescreen on boot.

I'm not seeing any interrupts for when the GPU is in use:

 PIN:  0  0  0  0  0
0  0  0  0  0  0
0  0  0  0  0   Posted-interrupt
notification event
 PIW:  0  0  0  0  0
0  0  0  0  0  0
0  0  0  0  0   Posted-interrupt wakeup
event


Here's my current XML:




  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  16384000
  16384000
  

  
  8
  4
  












  
  
hvm
/home/virtualguests/windows10/ovmf_code_x64.bin
/home/virtualguests/windows10/ovmf_vars_x64.bin

  
  




  
  
  


  


  
  

  
  




  
  destroy
  restart
  restart
  


  
  
/usr/bin/qemu-system-x86_64

  
  
  
  


  
  
  
  


  


  
  


  
  


  
  


  



  
  


  
  
  


  
  
  


  


  


  


  




  
  
  
  
  
  
  

   
  


  

   
  


  


  


  


  


  


  


  


  


  


  


  


  


  

  
  
  


  

  
  


  


On Mon, Apr 3, 2017 at 5:13 PM,  wrote:

> 
>
> Might perform better.
>
> Also, yes Hugepages might be helpful too.
>
> Also, would you mind observing whether interrupts are posted when the GPU
> is in use in pass-through?
>
> watch -d cat /proc/interrupts
>
> Look at the bottom for PIN (Posted-interrupt notification event) and PIW
> (Posted-interrupt wakeup event).
>
> Thanks!
>
> On 30 Mar 2017, at 15:46, Graham Neville wrote:
>
> Finally gotten to the bottom of this, this the help of your XML file I'm
> able to run with a Q35 setup, I'm also able to just passthrough the CPU
> features. Thanks.
>
> I discovered the issue with the guest crashing was actually due to my
> graphics card overheating! Noticed that it was hitting 99C and then guest
> would crash.
>
> The setup is pretty sweet now, I just want to change my drives from sata
> to virtio which I believe will be quicker. Then also investigate HugePages
>
> Here's my Kernel params:
>
> cat /proc/cmdline
> BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26
> rw quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
> isolcpus=0-7
>
> Here's my final XML file:
>
> 
>   Windows10
>   d45c3b5f-be8a-41e8-a22c-02e91c4c6245
>   8388608
>   8388608
>   4
>   
> 
> 
> 
> 
>   
>   
> hvm
> /home/
> virtualguests/windows10/ovmf_code_x64.bin
> /home/virtualguests/windows10/ovmf_vars_x64.bin
> 
>   
>   
> 
> 
> 
> 
>   
>   
>   
> 
> 
>   
> 
>   
>   
>   
> 
> 
> 
> 
>   
>   destroy
>   restart
>   restart
>   
> 
> 
>   
>   
> /usr/bin/qemu-system-x86_64
> 
>   
>   
>   
>   
> 
> 
>   
>   
>   
>   
> 
> 
>function='0x7'/>
> 
> 
>   
>function='0x0' multifunction='on'/>
> 
> 
>   
>function='0x1'/>
> 
> 
>   
>function='0x2'/>
> 
> 
>function='0x2'/>
> 
> 
> 
>   
>function='0x0'/>
> 
> 
>   
>   
>function='0x0'/>
> 
> 
>   
>   
>function='0x0'/>
> 
> 
>function='0x0'/>
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
> 
> 
>   
>   
>   
>   
>   
>   
> 
>
>   
> 
> 
>   
> 
>
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>function='0x0'/>
> 
> 
>   
> 
>   
>   
>   

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Graham Neville]

Finally gotten to the bottom of this, this the help of your XML file I'm
able to run with a Q35 setup, I'm also able to just passthrough the CPU
features. Thanks.

I discovered the issue with the guest crashing was actually due to my
graphics card overheating! Noticed that it was hitting 99C and then guest
would crash.

The setup is pretty sweet now, I just want to change my drives from sata to
virtio which I believe will be quicker. Then also investigate HugePages

Here's my Kernel params:

cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw
quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
isolcpus=0-7

Here's my final XML file:


  Windows10
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  8388608
  8388608
  4
  




  
  
hvm
/home/virtualguests/windows10/ovmf_code_x64.bin
/home/virtualguests/windows10/ovmf_vars_x64.bin

  
  




  
  
  


  

  
  
  




  
  destroy
  restart
  restart
  


  
  
/usr/bin/qemu-system-x86_64

  
  
  
  


  
  
  
  


  


  
  


  
  


  
  


  



  
  


  
  
  


  
  
  


  


  


  


  




  
  
  
  
  
  

   
  


  

   
  


  


  


  


  


  


  


  


  


  


  


  

  
  
  


  

  
  


  

  




On Thu, Mar 30, 2017 at 6:04 AM, Graham Neville 
wrote:

> I am using a ROM file which I dumped myself using GPU-Z, this was the only
> was I could get a display as well. With Q35 I do see TianoCore and then the
> Windows10 logo but the guest crashes straight after that. I may have to try
> and do a fresh install with Q35 to see if I can get further. Thanks for the
> XML, I'll try again tonight and report back.
>
>
> On 29 Mar 2017 11:33 p.m., "Steven Walter" 
> wrote:
>
> I didn't have good luck with Windows on BIOS or with Windows on i440.
> I pretty much had to use Q35 + OVMF.  You said you had trouble getting
> Windows to boot with Q35 + OVMF, did you at least get the TianoCore
> logo to come up on your screen?  If not you may need to dump and
> specify the ROM file for your video card.  For me OVMF would hang
> unless I did this.  Note that I'm using Q35 + OVMF, Opteron_G3 as CPU,
> specifying a ROM for the GPU, and running kvm_amd.avic=1.
>
> Here is my full guest XML file:
>
> 
>   WindowsTest
>   d45c3b5f-be8a-41e8-a22c-02e91c4c6245
>   8388608
>   8388608
>   8
>   
> hvm
> /usr/share/OVMF/
> OVMF_CODE.fd
> /var/lib/libvirt/qemu/nvram/WindowsTest_VARS.fd
> 
>   
>   
> 
> 
> 
>   
>   
>   
> 
>   
>   
>   
> 
> 
> 
> 
>   
>   destroy
>   restart
>   restart
>   
> 
> 
>   
>   
> /usr/bin/kvm-spice
> 
>   
>   
>   
>   
> 
> 
>   
>   
>   
>   
>   
> 
> 
>function='0x7'/>
> 
> 
>   
>function='0x0' multifunction='on'/>
> 
> 
>   
>function='0x1'/>
> 
> 
>   
>function='0x2'/>
> 
> 
>function='0x2'/>
> 
> 
> 
>   
>function='0x0'/>
> 
> 
>   
>   
>function='0x0'/>
> 
> 
>   
>   
>function='0x0'/>
> 
> 
>function='0x0'/>
> 
> 
>   
> 
> 
>   
> 
> 
>   
>   
> 
> 
>   
> 
> 
> 
> 
>   
> 
> 
>function='0x0'/>
> 
> 
>   
>function='0x0'/>
> 
> 
>   
> 
>   
>   
>function='0x0' multifunction='on'/>
> 
> 
>   
> 
>   
>function='0x1'/>
> 
> 
>   
> 
> 
>   
> 
> 
>function='0x0'/>
> 
>   
> 
>
> On Wed, Mar 29, 2017 at 5:44 PM, Graham Neville 
> wrote:
> > Thanks for the link. I've tried a number of things but still no further
> down
> > the line.
> >
> > I've tried the following one-by-one
> >
> > kvm_amd.avic=1
> > Properly isolated CPUs on host so they are exclusive to guest -
> isolcpus=0-7
> > Changed CPU host-passthrough to Opteron_G5 instead
> > Changed CPU to althon
> > Changed CPU to qemu64
> > kvm-amd.npt=0
> > iommu=pt
> > Disabled SMT in BIOS
> >
> > In using Arch Linux with Kernel 4.10.1.
> >
> > uname -a
> > Linux amdr7 4.10.5-1-ARCH #1 SMP 

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Steven Walter]

I didn't have good luck with Windows on BIOS or with Windows on i440.
I pretty much had to use Q35 + OVMF.  You said you had trouble getting
Windows to boot with Q35 + OVMF, did you at least get the TianoCore
logo to come up on your screen?  If not you may need to dump and
specify the ROM file for your video card.  For me OVMF would hang
unless I did this.  Note that I'm using Q35 + OVMF, Opteron_G3 as CPU,
specifying a ROM for the GPU, and running kvm_amd.avic=1.

Here is my full guest XML file:


  WindowsTest
  d45c3b5f-be8a-41e8-a22c-02e91c4c6245
  8388608
  8388608
  8
  
hvm
/usr/share/OVMF/OVMF_CODE.fd
/var/lib/libvirt/qemu/nvram/WindowsTest_VARS.fd

  
  



  
  
  

  
  
  




  
  destroy
  restart
  restart
  


  
  
/usr/bin/kvm-spice

  
  
  
  


  
  
  
  
  


  


  
  


  
  


  
  


  



  
  


  
  
  


  
  
  


  


  


  


  
  


  




  


  


  
  


  

  
  
  


  

  
  


  


  


  

  


On Wed, Mar 29, 2017 at 5:44 PM, Graham Neville  wrote:
> Thanks for the link. I've tried a number of things but still no further down
> the line.
>
> I've tried the following one-by-one
>
> kvm_amd.avic=1
> Properly isolated CPUs on host so they are exclusive to guest - isolcpus=0-7
> Changed CPU host-passthrough to Opteron_G5 instead
> Changed CPU to althon
> Changed CPU to qemu64
> kvm-amd.npt=0
> iommu=pt
> Disabled SMT in BIOS
>
> In using Arch Linux with Kernel 4.10.1.
>
> uname -a
> Linux amdr7 4.10.5-1-ARCH #1 SMP PREEMPT Wed Mar 22 14:42:03 CET 2017 x86_64
> GNU/Linux
>
> This is my kernel command line now:
>
> BOOT_IMAGE=/vmlinuz-linux root=UUID=bf69add2-e36f-453a-b92e-a4343ca20d26 rw
> quiet amd_iommu=on vfio-pci.ids=1002:67b1,1002:aac8 video=efifb:off
> amdgpu.msi=0 kvm_amd.avic=1 isolcpus=0-7 kvm-amd.npt=0 iommu=pt
>
> This is my full libvirt XML file for the VM:
>
>
> 
>   windows10
>   7b222825-fc7d-4a66-a72c-5876063752d5
>   8291456
>   8291456
>   4
>   
> 
> 
> 
> 
>   
>   
> hvm
>  readonly='yes'>/home/virtualguests/windows10/OVMF_CODE.fd
> /home/virtualguests/windows10/OVMF_VARS.fd
> 
>   
>  
> 
> 
> 
> 
>   
>   
>   
> 
> 
>   
> 
>   
>   
> 
>   
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
>   
> 
>   
>   
> 
> 
> 
> 
>   
>   destroy
>   restart
>   destroy
>   
> /usr/bin/qemu-system-x86_64
> 
>   
>   
>   
>function='0x0'/>
> 
> 
>   
>file='/home/virtualguests/windows10/Win10_1607_EnglishInternational_x64.iso'/>
>   
>   
>   
> 
> 
>   
>   
>   
> 
> 
> 
>   
>   
>   
>   
>   
>   
> 
> 
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
>   
> 
> 
> 
>   
> 
> 
>   
> 
>   
>   
>function='0x0' multifunction='on'/>
> 
> 
>   
> 
>   
>function='0x0'/>
> 
>
>   
> 
>
>
>
> When I have host-passthrough, Opteron_G5, althon or qemu64 CPUs configured I
> see a lot of these stack traces just appearing frequently and not just when
> the guest crashes, I see nothing when the guest crashes
>
> [ 2848.156709] [ cut here ]
> [ 2848.156719] WARNING: CPU: 0 PID: 1445 at arch/x86/kvm/svm.c:1484
> avic_vcpu_load+0x15a/0x180 [kvm_amd]
> [ 2848.156720] Modules linked in: vhost_net vhost macvtap macvlan tun nfsv3
> rpcsec_gss_krb5 nfsv4 dns_resolver nfs fscache hid_logitech_hidpp
> usb_serial_simple cdc_acm usbserial hid_logitech_dj cfg80211 bridge stp llc
> amdgpu sd_mod edac_mce_amd radeon edac_core kvm_amd kvm crct10dif_pclmul
> crc32_pclmul crc32c_intel ghash_clmulni_intel pcbc ppdev ttm
> snd_hda_codec_realtek snd_hda_codec_generic drm_kms_helper aesni_intel drm
> btusb snd_hda_intel nls_iso8859_1 aes_x86_64 btrtl crypto_simd nls_cp437
> btbcm glue_helper syscopyarea btintel sysfillrect snd_hda_codec vfat r8169
> joydev sysimgblt fat fb_sys_fops 

Re: [vfio-users] Ryzen Primary GPU passthrough success and woes

[Steven Walter]

I  got a similar (though multi-GPU) setup working, which I wrote up
here: 
https://www.reddit.com/r/VFIO/comments/616xih/gpu_passthrough_with_msi_b350_tomahawk/

One thing that may help you is to enable AVIC (kvm_amd.avic=1).  What
I saw without AVIC was that things would work briefly (only a few
seconds for me) before interrupts would stop getting delivered.
Sounds like things are working better for you without AVIC than they
did for me, but perhaps the extra improvement in IRQ latency would fix
the hangs you get during intensive graphics operations.


On Tue, Mar 28, 2017 at 6:02 PM, Graham Neville  wrote:
> I've managed to get pci-e passthough working on a gigabyte gaming 3 matx MB
> and Ryzen 1700, no ACS patch, using only 1 GPU - AMD r9 290. However I'm
> facing a problem with the whole KVM setup and not sure what it's related to.
> For the Windows10 guest with the GPU passed through it crashes (guest only,
> host is fine) whenever I try anything graphics intensive, for example
> running Witcher3. Normal desktop is fine.
> Also my Linux guests are acting odd when I try to SSH to them, I notice that
> the SSH terminals just stop working randomly. And then there's the issue
> with very slow network throughout to both VMs. I have no idea what's going
> on. It used to work fine with my Intel setup. There's no logs in dmesg to
> show a problem either.
>
> I'm going to try Seabios instead of OVMF to see if I can stop the crashing.
>
> Any one having similar issues or anyone can advise?
>
>
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>



-- 
-Steven Walter 

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] Ryzen Primary GPU passthrough success and woes

[Graham Neville]

I've managed to get pci-e passthough working on a gigabyte gaming 3 matx MB
and Ryzen 1700, no ACS patch, using only 1 GPU - AMD r9 290. However I'm
facing a problem with the whole KVM setup and not sure what it's related
to. For the Windows10 guest with the GPU passed through it crashes (guest
only, host is fine) whenever I try anything graphics intensive, for example
running Witcher3. Normal desktop is fine.
Also my Linux guests are acting odd when I try to SSH to them, I notice
that the SSH terminals just stop working randomly. And then there's the
issue with very slow network throughout to both VMs. I have no idea what's
going on. It used to work fine with my Intel setup. There's no logs in
dmesg to show a problem either.

I'm going to try Seabios instead of OVMF to see if I can stop the crashing.

Any one having similar issues or anyone can advise?
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users