Re: [libvirt-users] Libvirt newer than 2.1.0 doesnt start up
On Oct 28, 2016 7:24 PM, "Rene Pasing"wrote: > > On 10/28/2016 05:59 PM, Laine Stump wrote: > > Sigh. I am currently hanging my head in shame :-( > > > > See: > > https://www.redhat.com/archives/libvir-list/2016-October/msg01281.html > > > > My only excuse is that the task was too simple, and I've come to rely > > on reviewers too much so I was lazy and inattentive. > > > > Again, :-( > > Just to give short feedback: I just applied your patch to the official > 2.3.0 release, built it, installed it and yeah: That fixed my problem! > It works like a charm now. I pushed the patch upstream earlier today, so it will be in 2.4.0, which will be released next week. > > Thank you very much :) And thanks for taking the time to report the problem, and test the patch. ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] Libvirt newer than 2.1.0 doesnt start up
On 10/28/2016 05:59 PM, Laine Stump wrote: > Sigh. I am currently hanging my head in shame :-( > > See: > https://www.redhat.com/archives/libvir-list/2016-October/msg01281.html > > My only excuse is that the task was too simple, and I've come to rely > on reviewers too much so I was lazy and inattentive. > > Again, :-( Just to give short feedback: I just applied your patch to the official 2.3.0 release, built it, installed it and yeah: That fixed my problem! It works like a charm now. Thank you very much :) ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] sttic vnet device for guest
28.10.2016 23:32, Michal Privoznik пишет: On my host node i using system created bridge. example brctl show br1 bridge name bridge id STP enabled interfaces br1 8000.0025907925d3 no eth1 vnet0 vnet2 vnet3 vnetN - guest net adapter, It added to bridge at guest's node started. bridge defined as == internal == But in guest config xml vnet[0-3] nod defined === === >> Can i assign static vnet* device for some guests? > What do you mean? > vnet* devices are created by libvirt when a domain is being started (or > on device hotplug). In general, unless all devices would be statically > allocated, it would be impossible to guarantee certain vnet name. > > However, what you can do is to create the device yourself and then just > tell libvirt to use it: > > > > > > But most likely, the problem you are trying to solve looks for a > different solution. If you need the device name in order to set up some > environment (e.g. apply some FW rules on the device), we have network > hooks and domain hooks - user defined scripts that are run by libvirt on > various events (e.g. domain startup, device hotplug, etc.). You should > consider those. > > Michal > ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
[libvirt-users] sttic vnet device for guest
Can i assign static vnet* device for some guests? ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] [Qemu-devel] pci-assign fails with read error on config-space file
On Fri, 28 Oct 2016 11:25:55 -0400 Laine Stumpwrote: > On 10/28/2016 07:28 AM, Henning Schild wrote: > > Hey, > > > > i am running an unusual setup where i assign pci devices behind the > > back of libvirt. I have two options to do that: > > 1. a wrapper script for qemu that takes care of suid-root and appends > > arguments for pci-assign > > 2. virsh qemu-monitor-command ... 'device_add pci-assign...' > > With any reasonably modern version of Linux/qemu/libvirt, you should not > be using pci-assign, but should use vfio-pci instead. pci-assign is old, > unmaintained, and deprecated (and any other bad words you can think of). > > Also, have you done anything to lock the guest's memory in host RAM? > This is necessary so that the source/destination of DMA reads/writes is > always present. It is done automatically by libvirt as required *when > libvirt knows that a device is being assigned to the guest*, but if > you're going behind libvirt's back, you need to take care of that > yourself (or alternately, don't go behind libvirt's back, which is the > greatly preferred alternative!) Note that pci-assign doesn't care about user locked memory limits, so that much is not required for this deprecated use case, but I fully agree that going behind libvirt's back is completely unadvised and pci-assign is deprecated and likely broken. Maybe we should even consider removing it in the QEMU2.8 release cycle. Use vfio-pci and use the mechanisms provided in libvirt to attach the device to the VM, if these don't work, file bugs and improve the environment to meet your needs rather than working around it. I can't figure out from the original report what specifically about the environment prevents use of libvirt entries. Thanks, Alex > > > > I know i should probably not be doing this, > > > Yes, that is a serious understatement :-) And I suspect that it isn't > necessary. > > > > it is a workaround to > > introduce fine-grained pci-assignment in an openstack setup, where > > vendor and device id are not enough to pick the right device for a vm. > > libvirt selects the device according to its PCI address, not vendor and > device id. Is that not "fine-grained" enough? (And does OpenStack not > let you select devices based on their PCI address?) > > > > > In both cases qemu will crash with the following output: > > > >> qemu: hardware error: pci read failed, ret = 0 errno = 22 > > followed by the usual machine state dump. With strace i found it to be > > a failing read on the config space file of my device. > > /sys/bus/pci/devices/:xx:xx.x/config > > A few reads out of that file succeeded, as well as accesses on vendor > > etc. > > > > Manually launching a qemu with the pci-assign works without a problem, > > so i "blame" libvirt and the cgroup environment the qemu ends up in. > > So i put a bash into the exact same cgroup setup - next to a running > > qemu, expecting a dd or hexdump on the config-space file to fail. But > > from that bash i can read the file without a problem. > > > > Has anyone seen that problem before? > > No, because nobody else (that I've ever heard) is doing what you are > doing. You're going around behind the back of libvirt (and OpenStack) > to do device assignment with a method that was replaced with something > newer/better/etc about 3 years ago, and in the process are likely > missing a lot of the details that would otherwise be automatically > handled by libvirt. > > > > Right now i do not know what i > > am missing, maybe qemu is hitting some limits configured for the > > cgroups or whatever. I can not use pci-assign from libvirt, but if i > > did would it configure cgroups in a different way or relax some limits? > > > > What would be a good next step to debug that? Right now i am looking at > > kernel event traces, but the machine is pretty big and so is the trace. > > > My recommendation would be this: > > 1) look at OpenStack to see if it allows selecting the device to assign > by PCI address. If so, use that (it will just tell libvirt "assign this > device", and libvirt will automatically use VFIO for the device > assignment if it's available (which it will be)) > > 2) if (1) is a deadend (i.e. OpenStack doesn't allow you to select based > on PCI address), use your "sneaky backdoor method" to do "virsh > attach-device somexmlfile.xml", where somexmlfile.xml has a proper > element to select and assign the host device you want. Again, > libvirt will automatically figure out if VFIO can be used, and will > properly setup everything necessary related to cgroups, locked memory, etc. > > > > > > That assignment used to work and i do not know how it broke, i have > > tried combinations of several kernels, versions of libvirt and qemu. > > (kernel 3.18 and 4.4, libvirt 1.3.2 and 2.0.0, and qemu 2.2.1 and 2.7) > > All combinations show the same problem, even the ones that work on > > other machines. So when
Re: [libvirt-users] pci-assign fails with read error on config-space file
On 10/28/2016 07:28 AM, Henning Schild wrote: Hey, i am running an unusual setup where i assign pci devices behind the back of libvirt. I have two options to do that: 1. a wrapper script for qemu that takes care of suid-root and appends arguments for pci-assign 2. virsh qemu-monitor-command ... 'device_add pci-assign...' With any reasonably modern version of Linux/qemu/libvirt, you should not be using pci-assign, but should use vfio-pci instead. pci-assign is old, unmaintained, and deprecated (and any other bad words you can think of). Also, have you done anything to lock the guest's memory in host RAM? This is necessary so that the source/destination of DMA reads/writes is always present. It is done automatically by libvirt as required *when libvirt knows that a device is being assigned to the guest*, but if you're going behind libvirt's back, you need to take care of that yourself (or alternately, don't go behind libvirt's back, which is the greatly preferred alternative!) I know i should probably not be doing this, Yes, that is a serious understatement :-) And I suspect that it isn't necessary. it is a workaround to introduce fine-grained pci-assignment in an openstack setup, where vendor and device id are not enough to pick the right device for a vm. libvirt selects the device according to its PCI address, not vendor and device id. Is that not "fine-grained" enough? (And does OpenStack not let you select devices based on their PCI address?) In both cases qemu will crash with the following output: qemu: hardware error: pci read failed, ret = 0 errno = 22 followed by the usual machine state dump. With strace i found it to be a failing read on the config space file of my device. /sys/bus/pci/devices/:xx:xx.x/config A few reads out of that file succeeded, as well as accesses on vendor etc. Manually launching a qemu with the pci-assign works without a problem, so i "blame" libvirt and the cgroup environment the qemu ends up in. So i put a bash into the exact same cgroup setup - next to a running qemu, expecting a dd or hexdump on the config-space file to fail. But from that bash i can read the file without a problem. Has anyone seen that problem before? No, because nobody else (that I've ever heard) is doing what you are doing. You're going around behind the back of libvirt (and OpenStack) to do device assignment with a method that was replaced with something newer/better/etc about 3 years ago, and in the process are likely missing a lot of the details that would otherwise be automatically handled by libvirt. Right now i do not know what i am missing, maybe qemu is hitting some limits configured for the cgroups or whatever. I can not use pci-assign from libvirt, but if i did would it configure cgroups in a different way or relax some limits? What would be a good next step to debug that? Right now i am looking at kernel event traces, but the machine is pretty big and so is the trace. My recommendation would be this: 1) look at OpenStack to see if it allows selecting the device to assign by PCI address. If so, use that (it will just tell libvirt "assign this device", and libvirt will automatically use VFIO for the device assignment if it's available (which it will be)) 2) if (1) is a deadend (i.e. OpenStack doesn't allow you to select based on PCI address), use your "sneaky backdoor method" to do "virsh attach-device somexmlfile.xml", where somexmlfile.xml has a proper element to select and assign the host device you want. Again, libvirt will automatically figure out if VFIO can be used, and will properly setup everything necessary related to cgroups, locked memory, etc. That assignment used to work and i do not know how it broke, i have tried combinations of several kernels, versions of libvirt and qemu. (kernel 3.18 and 4.4, libvirt 1.3.2 and 2.0.0, and qemu 2.2.1 and 2.7) All combinations show the same problem, even the ones that work on other machines. So when it comes to software versions the problem could well be caused by a software update of another component, that i got with the package manager and did not compile myself. It is a debian 8.6 with all recent updates installed. My guess would be that systemd could have an influence on cgroups or limits causing such a problem. That you would need to think of such things points out that your current setup is fragile and ultimately unmaintainable. Please consider "coloring inside the lines" :-) (We'd be happy to help if there are any hangups along the way, either on the libvirt-users mailing list or in the #virt channel on irc.oftc.net). ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
[libvirt-users] pci-assign fails with read error on config-space file
Hey, i am running an unusual setup where i assign pci devices behind the back of libvirt. I have two options to do that: 1. a wrapper script for qemu that takes care of suid-root and appends arguments for pci-assign 2. virsh qemu-monitor-command ... 'device_add pci-assign...' I know i should probably not be doing this, it is a workaround to introduce fine-grained pci-assignment in an openstack setup, where vendor and device id are not enough to pick the right device for a vm. In both cases qemu will crash with the following output: > qemu: hardware error: pci read failed, ret = 0 errno = 22 followed by the usual machine state dump. With strace i found it to be a failing read on the config space file of my device. /sys/bus/pci/devices/:xx:xx.x/config A few reads out of that file succeeded, as well as accesses on vendor etc. Manually launching a qemu with the pci-assign works without a problem, so i "blame" libvirt and the cgroup environment the qemu ends up in. So i put a bash into the exact same cgroup setup - next to a running qemu, expecting a dd or hexdump on the config-space file to fail. But from that bash i can read the file without a problem. Has anyone seen that problem before? Right now i do not know what i am missing, maybe qemu is hitting some limits configured for the cgroups or whatever. I can not use pci-assign from libvirt, but if i did would it configure cgroups in a different way or relax some limits? What would be a good next step to debug that? Right now i am looking at kernel event traces, but the machine is pretty big and so is the trace. That assignment used to work and i do not know how it broke, i have tried combinations of several kernels, versions of libvirt and qemu. (kernel 3.18 and 4.4, libvirt 1.3.2 and 2.0.0, and qemu 2.2.1 and 2.7) All combinations show the same problem, even the ones that work on other machines. So when it comes to software versions the problem could well be caused by a software update of another component, that i got with the package manager and did not compile myself. It is a debian 8.6 with all recent updates installed. My guess would be that systemd could have an influence on cgroups or limits causing such a problem. regards, Henning ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users