Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Alex Williamson
On Wed, Oct 12, 2016 at 11:50 AM, Bronek Kozicki  wrote:

> On 12/10/2016 18:04, Alex Williamson wrote:
> > On Wed, Oct 12, 2016 at 10:51 AM, Ethan Thomas  > > wrote:
> >
> > I've been using a SuperMicro X8DTH-iF for quite some time with no
> > problems. However it's worth noting that with some generations of
> > multi-cpu boards the PCI-E lanes and ram may be associated with a
> > specific CPU, so you may need to adjust which slots and cores you
> > associate with a particular VM for best performance.
> >
> >
> > I would go so far as to say this is true of any modern multi-socket
> > system, it's called NUMA, Non-Uniform Memory Access. You can use tools
> > like lstopo to identify the locality of memory, devices, and
> > processors. Using memory and CPU from the correct node is important for
> > an VM, and an assigned device should be an extra pull towards the I/O
> > local node.
> ‎
> Hi Alex
>
> Memory locality is one thing I totally forgot about when setting up my
> VMs. Do you have any example how to reserve huge pages on a specific
> node via sysctl, and refer to it later in libvirt configuration?
> Currently I only use this:
>
> ~ # cat /etc/sysctl.d/80-hugepages.conf
> # Reserve this many 2MB pages for virtual machines
> vm.nr_hugepages = 28000
>

It's been a while since I've actively played with this, but I believe part
of the key is to configure hugepages via sysfs, not /proc/sys, which sysctl
does.  IIRC, /proc/sys/vm/nr_hugepages will by default do round-robin
allocation between nodes.  Instead you want to
use /sys/devices/system/node/node#/hugepages/ (replace # with node
number).  This allows you to specifically allocate per node and gives you
selection of what hugepage size to allocate.


> ~ # grep hugepages /etc/mtab
> hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0
>
> ~ # grep hugepages /etc/libvirt/qemu.conf | grep -vE "^#"
> hugetlbfs_mount = "/dev/hugepages"
>
> ~ # virsh dumpxml lublin-vfio1 | head -11
> 
> lublin-vfio1
> bc578734-6a43-4fda-9b19-e43225007a83
> 16777216
> 16777216
> 
> 
> 
> 
> 
> 
>
> ... which entirely ignores memory locality. TIA!


Libvirt has sections in their domain xml specification talking about numa
configurations:
http://libvirt.org/formatdomain.html#elementsNUMATuning

For a simple configuration where the VM fits within a host numa node, you'd
want to add something like:

  

  

Which should allocate all VM memory from numa node 1.  Use the information
in the sysfs paths above to verify that free hugepage count is going down
on the intended node.  If your VM spans NUMA nodes, things get a lot more
complicated and you'll need to fiddle with memnodes here and nodesets in
the  setup.
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users


Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Bronek Kozicki
On 12/10/2016 18:04, Alex Williamson wrote:
> On Wed, Oct 12, 2016 at 10:51 AM, Ethan Thomas  > wrote:
>
> I've been using a SuperMicro X8DTH-iF for quite some time with no
> problems. However it's worth noting that with some generations of
> multi-cpu boards the PCI-E lanes and ram may be associated with a
> specific CPU, so you may need to adjust which slots and cores you
> associate with a particular VM for best performance.
>
>
> I would go so far as to say this is true of any modern multi-socket
> system, it's called NUMA, Non-Uniform Memory Access. You can use tools
> like lstopo to identify the locality of memory, devices, and
> processors. Using memory and CPU from the correct node is important for
> an VM, and an assigned device should be an extra pull towards the I/O
> local node.
‎
Hi Alex

Memory locality is one thing I totally forgot about when setting up my 
VMs. Do you have any example how to reserve huge pages on a specific 
node via sysctl, and refer to it later in libvirt configuration? 
Currently I only use this:

~ # cat /etc/sysctl.d/80-hugepages.conf
# Reserve this many 2MB pages for virtual machines
vm.nr_hugepages = 28000

~ # grep hugepages /etc/mtab
hugetlbfs /dev/hugepages hugetlbfs rw,relatime 0 0

~ # grep hugepages /etc/libvirt/qemu.conf | grep -vE "^#"
hugetlbfs_mount = "/dev/hugepages"

~ # virsh dumpxml lublin-vfio1 | head -11

lublin-vfio1
bc578734-6a43-4fda-9b19-e43225007a83
16777216
16777216







... which entirely ignores memory locality. TIA!


B.



___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users


Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Ethan Thomas
I've been using a SuperMicro X8DTH-iF for quite some time with no problems.
However it's worth noting that with some generations of multi-cpu boards
the PCI-E lanes and ram may be associated with a specific CPU, so you may
need to adjust which slots and cores you associate with a particular VM for
best performance.

On Wed, Oct 12, 2016 at 8:41 AM, Jiri 'Ghormoon' Novak 
wrote:

> Hi,
>
> I'm considering rebuilding my setup from server components (mainly due
> to ECC ram), but I can't find much information on vga passtrough on
> server boards (supermicro, hp ...) to check if there are not issues eg.
> with stupid IRQ mapping like desktop boards sometimes have.
>
> is there any list of tested server boards? or can anyone recommend some
> (preferably few years old generations, I'm gonna buy secondhand HW, new
> servers are not in my reach)?
>
> Thanks,
> Gh.
>
> ___
> vfio-users mailing list
> vfio-users@redhat.com
> https://www.redhat.com/mailman/listinfo/vfio-users
>
___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users


Re: [vfio-users] vga passthrough on server boards

2016-10-12 Thread Bronek Kozicki

On 12/10/2016 13:41, Jiri 'Ghormoon' Novak wrote:

Hi,

I'm considering rebuilding my setup from server components (mainly due
to ECC ram), but I can't find much information on vga passtrough on
server boards (supermicro, hp ...) to check if there are not issues eg.
with stupid IRQ mapping like desktop boards sometimes have.

is there any list of tested server boards? or can anyone recommend some
(preferably few years old generations, I'm gonna buy secondhand HW, new
servers are not in my reach)?


FWIW, SuperMicro X9DA7 works very nicely for me.


B.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users