[ovirt-users] Re: vGPU on ovirt 4.3

[Michal Skrivanek]

> On 27. 1. 2021, at 19:20, Thomas Hoberg  wrote:
> 
>> On Wed, Jan 27, 2021 at 9:14 AM > 
>> 
>> Ok, I think I found at least for Nvidia. You can follow what described for
>> RHV:
>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/...
>> 
>> In the same manual there are also instructions for vGPU.
>> 
>> There is also the guide for 4.3:
>> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
>> 
>> Gianluca
> 
> Again, that's the easy part that works just fine.
> It's afterwards when you try to install the Nvidia driver when there might be 
> trouble because the driver refused to load on GPUs that aren't permitted to 
> operate inside a VM, quite independent of the hypervisor (same issue with 
> VMware, VirtualBox, KVM).
> 
> For that there are tricks you'll find on the Internet on how to cheat the 
> Nvidia driver into believing it's running on a physical machine but on KVM 
> that implies editing the XML config... which in the case of oVirt seems to be 
> generated at startup.

yes. you can use that.

> 
> So I guess the hooking mechanism can be used to take care of that, but I've 
> not progressed to that stage myself, mostly because the GPUs I use in the 
> corporate lab are whitelisted by Nvidia.

shouldn’t be too hard, you can use any of the existing hooks (even old ones) as 
a starting point. I think (I haven’t looked at that for a long time) that 
smbios modification may be enough, so there’s [1] right away. If not, it should 
be straightforward to adapt

> 
> And just to give a little credit: Pass-through works very well with oVirt 
> 4.3, only moving (inactive) VMs from host-to-host is a little more involved,
> while GPU live-migration unfortunately is out of the question. That makes 
> perfect sense for NICs and storage adapters, for GPUs one might argue that 
> their state could be migrated, too. But Linux doesn't really understand 
> heterogenous accelerators yet, so oVirt can't do better.

it’s a generic PCI VFIO passthrough, there’s nothing specific about GPU. 
Migrateable GPUs are not generic and will require some work, mostly in 
kernel/drivers. nvidia is working on it, but it’s hard to say when that’s going 
to be ready, and as always i doubt they’ll make it available for older/lowend 
cards. But then again maybe with some tricks…:)

Thanks,
michal

[1] https://github.com/oVirt/vdsm/tree/master/vdsm_hooks/smbios

> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3DBQHTQDOZ6ASIK64OLZBGSEZRO623Y/
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/OEZOY7LWJWEJ5WSOECGWT6G6N5CXZ2N4/

[ovirt-users] Re: vGPU on ovirt 4.3

[Thomas Hoberg]

> On Wed, Jan 27, 2021 at 9:14 AM  
> 
> Ok, I think I found at least for Nvidia. You can follow what described for
> RHV:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/...
> 
> In the same manual there are also instructions for vGPU.
> 
> There is also the guide for 4.3:
> https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/...
> 
> Gianluca

Again, that's the easy part that works just fine.
It's afterwards when you try to install the Nvidia driver when there might be 
trouble because the driver refused to load on GPUs that aren't permitted to 
operate inside a VM, quite independent of the hypervisor (same issue with 
VMware, VirtualBox, KVM).

For that there are tricks you'll find on the Internet on how to cheat the 
Nvidia driver into believing it's running on a physical machine but on KVM that 
implies editing the XML config... which in the case of oVirt seems to be 
generated at startup.

So I guess the hooking mechanism can be used to take care of that, but I've not 
progressed to that stage myself, mostly because the GPUs I use in the corporate 
lab are whitelisted by Nvidia.

And just to give a little credit: Pass-through works very well with oVirt 4.3, 
only moving (inactive) VMs from host-to-host is a little more involved, while 
GPU live-migration unfortunately is out of the question. That makes perfect 
sense for NICs and storage adapters, for GPUs one might argue that their state 
could be migrated, too. But Linux doesn't really understand heterogenous 
accelerators yet, so oVirt can't do better.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/S3DBQHTQDOZ6ASIK64OLZBGSEZRO623Y/

[ovirt-users] Re: vGPU on ovirt 4.3

[Gianluca Cecchi]

On Wed, Jan 27, 2021 at 9:14 AM  wrote:

> I would also be interested to know that :)
>

Ok, I think I found at least for Nvidia. You can follow what described for
RHV:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.4/html/setting_up_an_nvidia_gpu_for_a_virtual_machine_in_red_hat_virtualization/proc_nvidia_gpu_passthrough_nvidia_gpu_passthrough

In the same manual there are also instructions for vGPU.

There is also the guide for 4.3:
https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/setting_up_an_nvidia_gpu_for_a_virtual_machine_in_red_hat_virtualization/proc_nvidia_gpu_passthrough_nvidia_gpu_passthrough

Gianluca
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/TQWUGJJLS7ZCOVUDE7Y55MKGJLEQ6DGZ/

[ovirt-users] Re: vGPU on ovirt 4.3

[kim . kargaard]

I would also be interested to know that :)
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5K4J7HEIBVBUTMSMOODEOGFX4X2HYGMT/

[ovirt-users] Re: vGPU on ovirt 4.3

[Gianluca Cecchi]

On Tue, Jan 26, 2021 at 5:30 PM Michal Skrivanek <
michal.skriva...@redhat.com> wrote:

>
>
>
> why a hook? we have a “native” ovirt support for both GPU and vGPU since
> cca 4.1.
> There was a hook initially, but that’s no longer needed…since 4.2 iirc
> I think we also tested some Intel’s GTV-g card...
>
>
My suggestion  for the hook was only for GPU passthrough, not being P4000
supported for vGPU.
Good to know that GPU passthrough is supported/integrated. Do you have a
description of the workflow for doing it without a hook? Edit VM? Add
device?
Thanks,
Gianluca
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/5BRV2YNZU2U3DR4JXNT2YXH4L5QELXPN/

[ovirt-users] Re: vGPU on ovirt 4.3

[Michal Skrivanek]

> On 26. 1. 2021, at 12:50, Gianluca Cecchi  wrote:
> 
> On Tue, Jan 26, 2021 at 10:26 AM  > wrote:
> Thanks all for the feedback. 
> 
> When we bought the servers, they put some Nvidia Quadro P4000's in, but I 
> can't seem to find anything that says they support vGPU. 
> 
> I wasn't aware of the license server requirement for Nvidia, but the pricing 
> isn't too bad, so I can probably get that pushed through. Anyone aware of 
> which AMD cards are supported by oVirt? I have only found a list of Nvidia 
> cards. 
> 
> Thanks,
> 
> Kim
> 
> 
> 
> Hello,
> updated list of supported GPUs for vGPU functionality should be this one:
> https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html 
> 
> 
> and P4000 seems not to be present
> 
> So you could use the other supported method, with passthrough Gpu, following 
> what described here for qemu command line:
> https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#using-gpu-pass-through-red-hat-el-qemu-cli
>  
> 
> 
> and driving oVirt to do so with vdsm hook qemucmdline
> 


why a hook? we have a “native” ovirt support for both GPU and vGPU since cca 
4.1.
There was a hook initially, but that’s no longer needed…since 4.2 iirc
I think we also tested some Intel’s GTV-g card...



> [root@ov200 ~]# dnf info vdsm-hook-qemucmdline.noarch
> Last metadata expiration check: 2:48:59 ago on Tue 26 Jan 2021 09:58:06 AM 
> CET.
> Available Packages
> Name : vdsm-hook-qemucmdline
> Version  : 4.40.40
> Release  : 1.el8
> Architecture : noarch
> Size : 15 k
> Source   : vdsm-4.40.40-1.el8.src.rpm
> Repository   : ovirt-4.4
> Summary  : QEMU cmdline hook for VDSM
> URL  : http://www.ovirt.org/develop/developer-guide/vdsm/vdsm/ 
> 
> License  : GPLv2+
> Description  : Provides support for injecting QEMU cmdline via VDSM hook.
>  : It exploits libvirt's qemu:commandline facility available in 
> the
>  : qemu xml namespace.
> 
> [root@ov200 ~]# 
> 
> I donna if the doc here is up to date, not tested lately:
> https://www.ovirt.org/develop/developer-guide/vdsm/hook/qemucmdline.html 
> 
> 
> I think the persist command in case of node is not necessary with NG any more
> Developers could confirm and eventually point to the correct page for the 
> hook guide?
> 
> Gianluca
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct: 
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives: 
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/BKXIHMPQZQRNVBW2JS3YC3T4QMRT4JPF/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/3OB4CI2VEYTQDUJO4YI6POTASN3QI3MF/

[ovirt-users] Re: vGPU on ovirt 4.3

[Gianluca Cecchi]

On Tue, Jan 26, 2021 at 10:26 AM  wrote:

> Thanks all for the feedback.
>
> When we bought the servers, they put some Nvidia Quadro P4000's in, but I
> can't seem to find anything that says they support vGPU.
>
> I wasn't aware of the license server requirement for Nvidia, but the
> pricing isn't too bad, so I can probably get that pushed through. Anyone
> aware of which AMD cards are supported by oVirt? I have only found a list
> of Nvidia cards.
>
> Thanks,
>
> Kim
>
>

Hello,
updated list of supported GPUs for vGPU functionality should be this one:
https://docs.nvidia.com/grid/gpus-supported-by-vgpu.html

and P4000 seems not to be present

So you could use the other supported method, with passthrough Gpu,
following what described here for qemu command line:
https://docs.nvidia.com/grid/latest/grid-vgpu-user-guide/index.html#using-gpu-pass-through-red-hat-el-qemu-cli

and driving oVirt to do so with vdsm hook qemucmdline

[root@ov200 ~]# dnf info vdsm-hook-qemucmdline.noarch
Last metadata expiration check: 2:48:59 ago on Tue 26 Jan 2021 09:58:06 AM
CET.
Available Packages
Name : vdsm-hook-qemucmdline
Version  : 4.40.40
Release  : 1.el8
Architecture : noarch
Size : 15 k
Source   : vdsm-4.40.40-1.el8.src.rpm
Repository   : ovirt-4.4
Summary  : QEMU cmdline hook for VDSM
URL  : http://www.ovirt.org/develop/developer-guide/vdsm/vdsm/
License  : GPLv2+
Description  : Provides support for injecting QEMU cmdline via VDSM hook.
 : It exploits libvirt's qemu:commandline facility available in
the
 : qemu xml namespace.

[root@ov200 ~]#

I donna if the doc here is up to date, not tested lately:
https://www.ovirt.org/develop/developer-guide/vdsm/hook/qemucmdline.html

I think the persist command in case of node is not necessary with NG any
more
Developers could confirm and eventually point to the correct page for the
hook guide?

Gianluca
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/BKXIHMPQZQRNVBW2JS3YC3T4QMRT4JPF/

[ovirt-users] Re: vGPU on ovirt 4.3

[Shantur Rathore]

>
> I've been able to trick consumer GPUs into working with KVM on machines
> that I also use for oVirt, but oVirt itself is constructing the XML files
> for KVM on-the-fly, so you'd have to fiddle with the code that builds them:
> I haven't even found that yet.


You could use qemu_cmdline VDSM hook to provide custom qemu command line
params for your VM or you could write your own hook to change libvirt xml
before the VM is powered on.

On Mon, Jan 25, 2021 at 7:03 PM Alex McWhirter  wrote:

> IIRC oVirt 4.3 should have the basic hooks in place for mdev
> passthrough. For nvidia this mean you need the vgpu drivers and a
> license server. These licenses have a recurring cost.
>
> AMD's solution uses SR-IOV, and requires a custom kernel module that is
> not well tested YMMV.
>
> You can also passthrough entire cards in ovirt without any drivers,
> granted since the amount of GPU's you can stuff into a server is limited
> this is probably not ideal. Nvidia blocks this on consumer level cards.
>
> Intel also has a mdev solution that is built into mesa / the kernel
> already, we have used it with VCA cards in the past, but perhaps the new
> Intel GPU's support it as well? Intel is the only option that will allow
> you to feed the framebuffer data back into spice so you can use the
> ovirt console. All other options require you to create a remote session
> to the guest directly.
>
> On 2021-01-25 07:49, kim.karga...@noroff.no wrote:
> > Hi,
> >
> > We are looking at getting some GPU's for our servers and to use vGPU
> > passthrough so that our students can do some video renders on the
> > VM's. Does anyone have good experience with the Nvidia Quadro RTX6000
> > or RTX8000 and ovirt 4.3?
> >
> > Thanks.
> >
> > Kim
> > ___
> > Users mailing list -- users@ovirt.org
> > To unsubscribe send an email to users-le...@ovirt.org
> > Privacy Statement: https://www.ovirt.org/privacy-policy.html
> > oVirt Code of Conduct:
> > https://www.ovirt.org/community/about/community-guidelines/
> > List Archives:
> >
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/XF4LZOEAC2YTRH5LTL55YKA4BPRZLVZH/
> ___
> Users mailing list -- users@ovirt.org
> To unsubscribe send an email to users-le...@ovirt.org
> Privacy Statement: https://www.ovirt.org/privacy-policy.html
> oVirt Code of Conduct:
> https://www.ovirt.org/community/about/community-guidelines/
> List Archives:
> https://lists.ovirt.org/archives/list/users@ovirt.org/message/PY4ITOIWACUWHYLAVGOMMWIXUZYNDH5L/
>
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HLC5IMZUXQAKMKVZHGJGTL7TPU5EBLOM/

[ovirt-users] Re: vGPU on ovirt 4.3

[kim . kargaard]

Thanks all for the feedback. 

When we bought the servers, they put some Nvidia Quadro P4000's in, but I can't 
seem to find anything that says they support vGPU. 

I wasn't aware of the license server requirement for Nvidia, but the pricing 
isn't too bad, so I can probably get that pushed through. Anyone aware of which 
AMD cards are supported by oVirt? I have only found a list of Nvidia cards. 

Thanks,

Kim
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PYVDLRK6IS4EO4COA3CAE6EQZPOLNOGT/

[ovirt-users] Re: vGPU on ovirt 4.3

[Alex McWhirter]

IIRC oVirt 4.3 should have the basic hooks in place for mdev 
passthrough. For nvidia this mean you need the vgpu drivers and a 
license server. These licenses have a recurring cost.


AMD's solution uses SR-IOV, and requires a custom kernel module that is 
not well tested YMMV.


You can also passthrough entire cards in ovirt without any drivers, 
granted since the amount of GPU's you can stuff into a server is limited 
this is probably not ideal. Nvidia blocks this on consumer level cards.


Intel also has a mdev solution that is built into mesa / the kernel 
already, we have used it with VCA cards in the past, but perhaps the new 
Intel GPU's support it as well? Intel is the only option that will allow 
you to feed the framebuffer data back into spice so you can use the 
ovirt console. All other options require you to create a remote session 
to the guest directly.


On 2021-01-25 07:49, kim.karga...@noroff.no wrote:

Hi,

We are looking at getting some GPU's for our servers and to use vGPU
passthrough so that our students can do some video renders on the
VM's. Does anyone have good experience with the Nvidia Quadro RTX6000
or RTX8000 and ovirt 4.3?

Thanks.

Kim
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XF4LZOEAC2YTRH5LTL55YKA4BPRZLVZH/

___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/PY4ITOIWACUWHYLAVGOMMWIXUZYNDH5L/

[ovirt-users] Re: vGPU on ovirt 4.3

[Thomas Hoberg]

The main issue isn't oVirt, but Nvidia's drivers inside a virtual machine: You 
use 'unapproved' GPUs in what Nvidia drivers recognise as a VM, they'll refuse 
to load.

RTX series cards should be ok, I've tried K40 and P100 and they are just as 
fine with oVirt pass-through.
V100 tests are still outstanding, but I am glad Matthias is reporting success.

I've been able to trick consumer GPUs into working with KVM on machines that I 
also use for oVirt, but oVirt itself is constructing the XML files for KVM 
on-the-fly, so you'd have to fiddle with the code that builds them: I haven't 
even found that yet.

In terms of video rendering I've found the VirtualGL project 
(https://virtualgl.org/) rather fascinating, which allowed me to run the 
unengine singularity benchmark and "game" on a V100 GPU projected to a notebook 
hundreds of kilometers away...

The RTX series is supposed to enable partitioning of the GPUs, so several 
students can get a slice each: Never tried, don't know if that requries extra 
drivers etc. The target here is evidently remote CAD for aviation and defense, 
where money belts are quite a bit wider than in education.
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/HBYG6MM7GX5ILPO3MRBGKP2EDXTGXARB/

[ovirt-users] Re: vGPU on ovirt 4.3

[Matthias Leopold]

I can share experience with vGPUs on Tesla V100 PCIe 32GB GPUs, that are 
used for CUDA applications (but no rendering...)


Matthias

Am 25.01.21 um 13:49 schrieb kim.karga...@noroff.no:

Hi,

We are looking at getting some GPU's for our servers and to use vGPU 
passthrough so that our students can do some video renders on the VM's. Does 
anyone have good experience with the Nvidia Quadro RTX6000 or RTX8000 and ovirt 
4.3?

Thanks.

Kim
___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/XF4LZOEAC2YTRH5LTL55YKA4BPRZLVZH/


___
Users mailing list -- users@ovirt.org
To unsubscribe send an email to users-le...@ovirt.org
Privacy Statement: https://www.ovirt.org/privacy-policy.html
oVirt Code of Conduct: 
https://www.ovirt.org/community/about/community-guidelines/
List Archives: 
https://lists.ovirt.org/archives/list/users@ovirt.org/message/EKGMDB5RQ6XVB5N3ZC2CFITCPW6WLEAJ/