Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Alex Williamson 
On Wed, Oct 12, 2016 at 11:50 AM, Bronek Kozicki  wrote:

> On 12/10/2016 18:04, Alex Williamson wrote:
> > On Wed, Oct 12, 2016 at 10:51 AM, Ethan Thomas  > > wrote:
> >
> > I've been using a SuperMicro X8DTH-iF for quite some time with no
> > problems. However it's worth noting that with some generations of
> > multi-cpu boards the PCI-E lanes and ram may be associated with a
> > specific CPU, so you may need to adjust which slots and cores you
> > associate with a particular VM for best performance.
> >
> >
> > I would go so far as to say this is true of any modern multi-socket
> > system, it's called NUMA, Non-Uniform Memory Access. You can use tools
> > like lstopo to identify the locality of memory, devices, and
> > processors. Using memory and CPU from the correct node is important for
> > an VM, and an assigned device should be an extra pull towards the I/O
> > local node.
> ‎
> Hi Alex
>
> Memory locality is one thing I totally forgot about when setting up my
> VMs. Do you have any example how to reserve huge pages on a specific
> node via sysctl, and refer to it later in libvirt configuration?
> Currently I only use this:
>
> ~ # cat /etc/sysctl.d/80-hugepages.conf
> # Reserve this many 2MB pages for virtual machines
> vm.nr_hugepages = 28000
>

It's been a while since I've actively played with this, but I believe part
of the key is to configure hugepages via sysfs, not /proc/sys, which sysctl
does.  IIRC, /proc/sys/vm/nr_hugepages will by default do round-robin
allocation between nodes.  Instead you want to
use /sys/devices/system/node/node#/hugepages/ (replace # with node
number).  This allows you to specifically allocate per node and gives you
selection of what hugepage size to allocate.


> ~ # grep hugepages /etc/mtab
> hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
>
> ~ # grep hugepages /etc/libvirt/qemu.conf | grep -vE "^#"
> hugetlbfs_mount = "/dev/hugepages"
>
> ~ # virsh dumpxml lublin-vfio1 | head -11
> 
> lublin-vfio1
> bc578734-6a43-4fda-9b19-e43225007a83
> 16777216
> 16777216
> 
> 
> 
> 
> 
> 
>
> ... which entirely ignores memory locality. TIA!


Libvirt has sections in their domain xml specification talking about numa
configurations:
http://libvirt.org/formatdomain.html#elementsNUMATuning

For a simple configuration where the VM fits within a host numa node, you'd
want to add something like:

  

  

Which should allocate all VM memory from numa node 1.  Use the information
in the sysfs paths above to verify that free hugepage count is going down
on the intended node.  If your VM spans NUMA nodes, things get a lot more
complicated and you'll need to fiddle with memnodes here and nodesets in
the  setup.
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Bronek Kozicki 
On 12/10/2016 18:04, Alex Williamson wrote:
> On Wed, Oct 12, 2016 at 10:51 AM, Ethan Thomas  > wrote:
>
> I've been using a SuperMicro X8DTH-iF for quite some time with no
> problems. However it's worth noting that with some generations of
> multi-cpu boards the PCI-E lanes and ram may be associated with a
> specific CPU, so you may need to adjust which slots and cores you
> associate with a particular VM for best performance.
>
>
> I would go so far as to say this is true of any modern multi-socket
> system, it's called NUMA, Non-Uniform Memory Access. You can use tools
> like lstopo to identify the locality of memory, devices, and
> processors. Using memory and CPU from the correct node is important for
> an VM, and an assigned device should be an extra pull towards the I/O
> local node.
‎
Hi Alex

Memory locality is one thing I totally forgot about when setting up my 
VMs. Do you have any example how to reserve huge pages on a specific 
node via sysctl, and refer to it later in libvirt configuration? 
Currently I only use this:

~ # cat /etc/sysctl.d/80-hugepages.conf
# Reserve this many 2MB pages for virtual machines
vm.nr_hugepages = 28000

~ # grep hugepages /etc/mtab
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0

~ # grep hugepages /etc/libvirt/qemu.conf | grep -vE "^#"
hugetlbfs_mount = "/dev/hugepages"

~ # virsh dumpxml lublin-vfio1 | head -11

lublin-vfio1
bc578734-6a43-4fda-9b19-e43225007a83
16777216
16777216







... which entirely ignores memory locality. TIA!


B.



___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Ethan Thomas 
I've been using a SuperMicro X8DTH-iF for quite some time with no problems.
However it's worth noting that with some generations of multi-cpu boards
the PCI-E lanes and ram may be associated with a specific CPU, so you may
need to adjust which slots and cores you associate with a particular VM for
best performance.

On Wed, Oct 12, 2016 at 8:41 AM, Jiri 'Ghormoon' Novak 
wrote:

> Hi,
>
> I'm considering rebuilding my setup from server components (mainly due
> to ECC ram), but I can't find much information on vga passtrough on
> server boards (supermicro, hp ...) to check if there are not issues eg.
> with stupid IRQ mapping like desktop boards sometimes have.
>
> is there any list of tested server boards? or can anyone recommend some
> (preferably few years old generations, I'm gonna buy secondhand HW, new
> servers are not in my reach)?
>
> Thanks,
> Gh.
>
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vfio-pci late loading, built-in or initramfs

2016-10-12 Thread Alex Williamson 
On Tue, 11 Oct 2016 22:09:06 -0700
Muted Bytes  wrote:

> I have recently installed an AMD card for my host system, and my vm
> guest is also using an AMD card. On the host, I am using radeon driver
> but would like to prevent the guest getting radeon loaded. With
> pci-stub, with it built-in to the kernel, it would always grab pci
> devices before any other driver got the chance. However, with vfio-pci
> it seems to be loaded much later in the boot process, so radeon gets
> both cards before vfio-pci can bind it. As a result, I have tried
> building vfio and vfio-pci as modules and then including them in a
> Dracut-created initramfs as detailed at
> https://vfio.blogspot.com/2015/05/vfio-gpu-how-to-series-part-3-host.html.
> Even doing this, vfio-pci still gets loaded as late as it was built-in
> to the kernel, and so it does not get loaded before radeon (which is
> built-in).
> 
> Is there a way to get vfio-pci loaded earlier? Do I need to just build
> radeon as module to hope it gets loaded later also?

If radeon is built-in, you're causing your own problems.  There is no
way to get a module to load prior to a built-in driver.  Even ordering
of built-in drivers is based on link ordering.  This is part of why we
use modules.  Modules are a good thing.  I'd guess your only hope is to
have pci-stub built-in and hope the link order lets it probe the
devices first, then sort out later how to get the correct devices bound
to vfio-pci vs radeon.  Thanks,

Alex

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Bronek Kozicki 

On 12/10/2016 13:41, Jiri 'Ghormoon' Novak wrote:

Hi,

I'm considering rebuilding my setup from server components (mainly due
to ECC ram), but I can't find much information on vga passtrough on
server boards (supermicro, hp ...) to check if there are not issues eg.
with stupid IRQ mapping like desktop boards sometimes have.

is there any list of tested server boards? or can anyone recommend some
(preferably few years old generations, I'm gonna buy secondhand HW, new
servers are not in my reach)?


FWIW, SuperMicro X9DA7 works very nicely for me.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

[vfio-users] vga passthrough on server boards

2016-10-12 Thread Jiri 'Ghormoon' Novak 
Hi,

I'm considering rebuilding my setup from server components (mainly due
to ECC ram), but I can't find much information on vga passtrough on
server boards (supermicro, hp ...) to check if there are not issues eg.
with stupid IRQ mapping like desktop boards sometimes have.

is there any list of tested server boards? or can anyone recommend some
(preferably few years old generations, I'm gonna buy secondhand HW, new
servers are not in my reach)?

Thanks,
Gh.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Win10 Guest, stability is horrible and freeze quickly

2016-10-12 Thread Berillions 
I resolved my problem with both Nvidia GPU. Use Q35 instead of i440fx
helped me. :D

2016-10-08 15:16 GMT+02:00 Berillions :

> I don't understand what could eats the RAM of the host because i have 32GB
> Memory RAM. I have 16GB for the Host and 16GB (with Hugepages) for the
> Guest. My Windows_10.img is in an Ext2 partition.
>
> I try actually to use the IGD (Intel graphic card) to the Host and the
> GTX970 to the Guest and i have no problems. How is it possible to have
> conflict between both Nvidia card.
> I have only two solutions :
>
> 1- Use both Nvidia cards but only 2GB Memory RAM in virt-manager and i
> don't know if with 10GB of Memory swap in Win10 is enough to play at AAA
> games
>
> 2- Use IGD for Linux and the Nvidia for the Host.
>
> 2016-10-08 14:36 GMT+02:00 Bronek Kozicki :
>
>> Perhaps you have something eating into RAM of the host. I had my host
>> freezing when I forgot to limit memory utilization in the ZFS. It does not
>> matter what eats into your host RAM - since qemu is just a regular
>> userspace process, it's memory needs (ie those of the guest system) are not
>> "prioritized" as more essential than any other userspace process. If your
>> host does not have enough memory, then the more you allocate in qemu (or
>> libvirt) to the guest system, the more probability that qemu will run out
>> of RAM, effectively freezing the guest OS. At least, this is what it would
>> appear to be (it might be simply swapping, but that slows down guest
>> execution speed by orders of magnitude, so it is effectively frozen)
>>
>>
>> B.
>>
>> *From: *Berillions
>> *Sent: *Friday, 7 October 2016 20:00
>> *To: *Jayme Howard
>> *Cc: *mar...@schrodt.org; vfio-users
>> *Subject: *Re: [vfio-users] Win10 Guest, stability is horrible and
>> freeze quickly
>>
>> Very very very things. If i set more than 2GB Memory Ram in Virt-Manager
>> for the Guest, i have artefacts and freeze. If I set 2GB like actually, the
>> guest works perfectly.
>>
>> So i don't know if it will be good for gaming even if i set Windows swap
>> to 10GB ...
>>
>> 2016-10-07 19:43 GMT+02:00 Jayme Howard :
>>
>>> Put the 960 into the one that goes 4x if you're that worried about it,
>>> but it shouldn't be a huge deal.  I don't know if the 960 can even saturate
>>> a 4x.  It might be able to, but since it's a lower performance card anyway,
>>> it shouldn't really matter.
>>>
>>> On Fri, Oct 7, 2016 at 12:36 PM, Berillions 
>>> wrote:
>>>
 I have two PCI-E 3.0 16x slot,

 The first with the 960 and the second for the 970. if the second slot
 is used, his bandwidth will not be 16x but 4x.
 The others slots are only PCI-E 3.0 1x ...

 ___
 vfio-users mailing list
 vfio-users@redhat.com
 https://www.redhat.com/mailman/listinfo/vfio-users


>>>
>>
>>
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users