Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On 03/20/2017 07:37 PM, Erik Skultety wrote: On Mon, Mar 20, 2017 at 11:03:40AM +0100, Erik Skultety wrote: On Mon, Mar 20, 2017 at 03:27:19PM +0800, yonglihe wrote: tested v3, on the mdev-next branch: none-root: == 1. hacker the privilege operations sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME" sudo chown ubuntu:ubuntu /dev/vfio/0 RFC: i don't know the correct way to do such thing while build it from sources. updated me, thanks. I'm not sure what you're referring to, so could you be more specific, so we can work from there and find the right solution? while i try starting a non-privileged vm, there is 2 problem i run into: 1. the ulimit can not be changed cause the virsh user don't have that right to do so. 2. the /dev/vfio/0 (i got only one mdev), is does not own by the user. that's all I’m talking about. [...] rooted mode start libvirt-d trace: -- this trace shows occasionally while starting the libvirt-d, not every time. 2017-03-19 19:22:45.559+: 13104: info : libvirt version: 3.2.0 2017-03-19 19:22:45.559+: 13104: info : hostname: z-nuc-11.maas 2017-03-19 19:22:45.559+: 13104: error : qemuMonitorOpenUnix:367 : failed to connect to monitor socket: No such process 2017-03-19 19:22:45.562+: 13000: info : libvirt version: 3.2.0 2017-03-19 19:22:45.562+: 13000: info : hostname: z-nuc-11.maas 2017-03-19 19:22:45.562+: 13000: error : virNetSocketReadWire:1800 : End of file while reading data: Input/output error Hmm, the virNetSocketReadWire might go away with the recent libvirt patches (I rebased my branch onto current master) if you're using either virtlogd (which is the default in qemu.conf on RHEL if I'm not mistaken) or virtlockd, so try ^^The fix I mentioned would only make difference if all the daemons would log into the same file or to journal, but the log format above is different from what you'd see in the journal and I assume there is a dedicated log file for each daemon, so please ignore my post above. I'll still need complete logs and refer to attachment. the logfile is messed up. i will update you later. i'm going to try pull new patches, and retest it. Yongli He the configs though. Erik with the rebased branch. But I doubt the qemuMonitorOpenUnix bit would disappear with the mentioned libvirt fix. But I will need a fresh and complete daemon log to investigate further, as well as libvirtd.conf and qemu.conf (just in case), so that I can compare to my environment, since I'm not able to reproduce this. [...] NOTES: there is no traces under none root mode, though i don't think the trace is related to user privilege. fix me. It's indeed odd that you don't see the same behaviour when running qemu as root. As I said, I will need the daemon log and preferably the configs as well to investigate further. Thanks, Erik -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list # Master configuration file for the QEMU driver. # All settings described here are optional - if omitted, sensible # defaults are used. # Use of TLS requires that x509 certificates be issued. The default is # to keep them in /etc/pki/qemu. This directory must contain # # ca-cert.pem - the CA master certificate # server-cert.pem - the server certificate signed with ca-cert.pem # server-key.pem - the server private key # # and optionally may contain # # dh-params.pem - the DH params configuration file # #default_tls_x509_cert_dir = "/etc/pki/qemu" # The default TLS configuration only uses certificates for the server # allowing the client to verify the server's identity and establish # an encrypted channel. # # It is possible to use x509 certificates for authentication too, by # issuing a x509 certificate to every client who needs to connect. # # Enabling this option will reject any client who does not have a # certificate signed by the CA in /etc/pki/qemu/ca-cert.pem # #default_tls_x509_verify = 1 # # Libvirt assumes the server-key.pem file is unencrypted by default. # To use an encrypted server-key.pem file, the password to decrypt # the PEM file is required. This can be provided by creating a secret # object in libvirt and then to uncomment this setting to set the UUID # of the secret. # # NB This default all-zeros UUID will not work. Replace it with the # output from the UUID for the TLS secret from a 'virsh secret-list' # command and then uncomment the entry # #default_tls_x509_secret_uuid = "----" # VNC is configured to listen on 127.0.0.1 by default. # To make it listen on all public interfaces, uncomment # this next option. # # NB, strong recommendation to enable TLS + x509 certificate # verification when allowing public access # #vnc_listen = "0.0.0.0" # Enable this option to have VNC served over an automatically created # unix socket. This prevents unprivileged access from users on the # host machine, though most VNC clients do
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On Mon, Mar 20, 2017 at 11:03:40AM +0100, Erik Skultety wrote: > On Mon, Mar 20, 2017 at 03:27:19PM +0800, yonglihe wrote: > > > > tested v3, on the mdev-next branch: > > > > none-root: > > == > > 1. hacker the privilege operations > > sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME" > > sudo chown ubuntu:ubuntu /dev/vfio/0 > > > > RFC: i don't know the correct way to do such thing while build it from > > sources. updated me, thanks. > > I'm not sure what you're referring to, so could you be more specific, so we > can > work from there and find the right solution? > > [...] > > > rooted mode > > > > > > > > start libvirt-d trace: > > -- > > > > this trace shows occasionally while starting the libvirt-d, not every time. > > > > 2017-03-19 19:22:45.559+: 13104: info : libvirt version: 3.2.0 > > 2017-03-19 19:22:45.559+: 13104: info : hostname: z-nuc-11.maas > > 2017-03-19 19:22:45.559+: 13104: error : qemuMonitorOpenUnix:367 : > > failed to connect to monitor socket: No such process > > 2017-03-19 19:22:45.562+: 13000: info : libvirt version: 3.2.0 > > 2017-03-19 19:22:45.562+: 13000: info : hostname: z-nuc-11.maas > > 2017-03-19 19:22:45.562+: 13000: error : virNetSocketReadWire:1800 : End > > of file while reading data: Input/output error > > Hmm, the virNetSocketReadWire might go away with the recent libvirt patches (I > rebased my branch onto current master) if you're using either virtlogd (which > is the default in qemu.conf on RHEL if I'm not mistaken) or virtlockd, so try ^^The fix I mentioned would only make difference if all the daemons would log into the same file or to journal, but the log format above is different from what you'd see in the journal and I assume there is a dedicated log file for each daemon, so please ignore my post above. I'll still need complete logs and the configs though. Erik > with the rebased branch. But I doubt the qemuMonitorOpenUnix bit would > disappear with the mentioned libvirt fix. But I will need a fresh and complete > daemon log to investigate further, as well as libvirtd.conf and qemu.conf > (just > in case), so that I can compare to my environment, since I'm not able to > reproduce this. > > [...] > > > NOTES: > >there is no traces under none root mode, though i don't think the trace > > is related to user privilege. fix me. > > It's indeed odd that you don't see the same behaviour when running qemu as > root. As I said, I will need the daemon log and preferably the configs as well > to investigate further. > > Thanks, > Erik > > -- > libvir-list mailing list > libvir-list@redhat.com > https://www.redhat.com/mailman/listinfo/libvir-list -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On Mon, Mar 20, 2017 at 03:27:19PM +0800, yonglihe wrote: > > tested v3, on the mdev-next branch: > > none-root: > == > 1. hacker the privilege operations > sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME" > sudo chown ubuntu:ubuntu /dev/vfio/0 > > RFC: i don't know the correct way to do such thing while build it from > sources. updated me, thanks. I'm not sure what you're referring to, so could you be more specific, so we can work from there and find the right solution? [...] > rooted mode > > > > start libvirt-d trace: > -- > > this trace shows occasionally while starting the libvirt-d, not every time. > > 2017-03-19 19:22:45.559+: 13104: info : libvirt version: 3.2.0 > 2017-03-19 19:22:45.559+: 13104: info : hostname: z-nuc-11.maas > 2017-03-19 19:22:45.559+: 13104: error : qemuMonitorOpenUnix:367 : > failed to connect to monitor socket: No such process > 2017-03-19 19:22:45.562+: 13000: info : libvirt version: 3.2.0 > 2017-03-19 19:22:45.562+: 13000: info : hostname: z-nuc-11.maas > 2017-03-19 19:22:45.562+: 13000: error : virNetSocketReadWire:1800 : End > of file while reading data: Input/output error Hmm, the virNetSocketReadWire might go away with the recent libvirt patches (I rebased my branch onto current master) if you're using either virtlogd (which is the default in qemu.conf on RHEL if I'm not mistaken) or virtlockd, so try with the rebased branch. But I doubt the qemuMonitorOpenUnix bit would disappear with the mentioned libvirt fix. But I will need a fresh and complete daemon log to investigate further, as well as libvirtd.conf and qemu.conf (just in case), so that I can compare to my environment, since I'm not able to reproduce this. [...] > NOTES: >there is no traces under none root mode, though i don't think the trace > is related to user privilege. fix me. It's indeed odd that you don't see the same behaviour when running qemu as root. As I said, I will need the daemon log and preferably the configs as well to investigate further. Thanks, Erik -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On 2017年03月16日 22:41, Erik Skultety wrote: [2] 2005 ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ *** start libvirt-d 2017-03-09 19:04:57.211+: 2059: info : libvirt version: 3.1.0 2017-03-09 19:04:57.211+: 2059: info : hostname: z-nuc-11.maas 2017-03-09 19:04:57.211+: 2059: error : qemuMonitorOpenUnix:367 : failed to connect to monitor socket: No such process 2017-03-09 19:04:57.213+: 2059: error : virMediatedDeviceGetIOMMUGroupDev:153 : internal error: IOMMU group file /sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be/iommu_group is not a symlink When I saw this error message for the first time in the original thread, I got confused, since this just checks whether the symlink exists, if it doesn't, the vfio device probably also doesn't exist (but take this with a grain of salt, I haven't investigated that deep) and libvirt needs it to pass it onto qemu command line. I hit this issue once by accident in the past and at that time I didn't understand what caused it, but after a reboot it was gone. So seeing it here it caught my eye and I investigated it last week. What I found out was that it's caused by the vfio-mdev module not being loaded automatically as a dependency. I solved it by autoloading the module on system boot. So this is not a libvirt issue, but just for a reference, there is a BZ on this [1]. it's does not show up every time. might be my bad. 2017-03-09 19:04:57.213+: 2003: info : libvirt version: 3.1.0 2017-03-09 19:04:57.213+: 2003: info : hostname: z-nuc-11.maas 2017-03-09 19:04:57.213+: 2003: error : virNetSocketReadWire:1800 : End of file while reading data: Input/output error I suppose this corresponds to the problem above, do you hit this error if you work around the vfio-mdev module problem described above? the screen call trace while start the VM (same for Ubuntu, Win10 etc) == ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : Domain id=1 name='vgpu-ubuntu' uuid=972b5e38-0437-11e7-8f97-d36dba74552d is tainted: high-privileges 2017-03-09 19:06:50.819+: 2204: info : libvirt version: 3.1.0 2017-03-09 19:06:50.819+: 2232: warning : virDomainAuditHostdev:456 : Unexpected hostdev type while encoding audit message: 4 This one's interesting, again, are you able to hit the error when you work around the missing vfio-mdev module? I'll have a look at this if you actually can hit the error, even if the XML is correct. the module should be there, the domain is running and virtual gpu is working. ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ sudo lsmod| grep mdev vfio_mdev 16384 1 mdev 20480 2 kvmgt,vfio_mdev vfio 28672 6 vfio_iommu_type1,kvmgt,vfio_mdev on the v3, the trace still there. refer to my "test by" email on mdev-next. Regards Yongli He I posted v3 of the series and also created a new branch 'mdev-next' on my github [2]. I dropped the attribute 'type' from the source address element, so follow the example in the updated docs. Thanks for giving it a try. Erik [1] https://bugzilla.redhat.com/show_bug.cgi?id=1420572 [2] https://github.com/eskultety/libvirt/commits/mdev-next -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
tested v3, on the mdev-next branch: none-root: == 1. hacker the privilege operations sudo sh -c "ulimit -l 3074424832 && exec su $LOGNAME" sudo chown ubuntu:ubuntu /dev/vfio/0 RFC: i don't know the correct way to do such thing while build it from sources. updated me, thanks. 2. myvirsh -c qemu:///session #define ./libvirt/vgpu-win10.xml Domain vgpu-win10 defined from ./libvirt/vgpu-win10.xml #start vgpu-win10 Domain vgpu-win10 started 3.ps aux | grep qemu ubuntu3262 141 12.3 2929432 2017172 ? SLl 23:58 0:51 /usr/bin/qemu-system-x86_64 -name guest=vgpu-win10,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/home/ubuntu/.config/libvirt/qemu/lib/domain-1-vgpu-win10/master-key.aes -machine pc-i440fx-2.3,accel=kvm,usb=off,dump-guest-core=off -m 1908 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid 916c5c36-0437-11e7-a23d-830ed1295d00 -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/home/ubuntu/.config/libvirt/qemu/lib/domain-1-vgpu-win10/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/home/ubuntu/vgpu-meta/libvirt-stage/win10-64.qcow2,format=qcow2,if=none,id=drive-ide0-0-0,cache=none,aio=native -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:0 -device cirrus-vga,id=video0,bus=pci.0,addr=0x2 -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be,bus=pci.0,addr=0x4 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3 -msg timestamp=on ubuntu3276 0.0 0.0 10468 2216 pts/0S+ 23:59 0:00 grep --color=auto qemu rooted mode start libvirt-d trace: -- this trace shows occasionally while starting the libvirt-d, not every time. 2017-03-19 19:22:45.559+: 13104: info : libvirt version: 3.2.0 2017-03-19 19:22:45.559+: 13104: info : hostname: z-nuc-11.maas 2017-03-19 19:22:45.559+: 13104: error : qemuMonitorOpenUnix:367 : failed to connect to monitor socket: No such process 2017-03-19 19:22:45.562+: 13000: info : libvirt version: 3.2.0 2017-03-19 19:22:45.562+: 13000: info : hostname: z-nuc-11.maas 2017-03-19 19:22:45.562+: 13000: error : virNetSocketReadWire:1800 : End of file while reading data: Input/output error start domian trace: -- some time there is trace while starting the domain. 2017-03-19 19:22:50.912+: 13034: warning : qemuDomainObjTaint:4113 : Domain id=3 name='vgpu-win10' uuid=916c5c36-0437-11e7-a23d-830ed1295d00 is tainted: high-privileges 2017-03-19 19:22:51.859+: 13000: error : virNetSocketReadWire:1800 : End of file while reading data: Input/output error 2017-03-19 19:22:51.859+: 13034: warning : virDomainAuditHostdev:456 : Unexpected hostdev type while encoding audit message: 4 Domain vgpu-win10 started NOTES: there is no traces under none root mode, though i don't think the trace is related to user privilege. fix me. Regards Yongli He On 2017年03月16日 22:41, Erik Skultety wrote: [2] 2005 ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ *** start libvirt-d 2017-03-09 19:04:57.211+: 2059: info : libvirt version: 3.1.0 2017-03-09 19:04:57.211+: 2059: info : hostname: z-nuc-11.maas 2017-03-09 19:04:57.211+: 2059: error : qemuMonitorOpenUnix:367 : failed to connect to monitor socket: No such process 2017-03-09 19:04:57.213+: 2059: error : virMediatedDeviceGetIOMMUGroupDev:153 : internal error: IOMMU group file /sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be/iommu_group is not a symlink When I saw this error message for the first time in the original thread, I got confused, since this just checks whether the symlink exists, if it doesn't, the vfio device probably also doesn't exist (but take this with a grain of salt, I haven't investigated that deep) and libvirt needs it to pass it onto qemu command line. I hit this issue once by accident in the past and at that time I didn't understand what caused it, but after a reboot it was gone. So seeing it here it caught my eye and I investigated it last week. What I found out was that it's caused by the vfio-mdev module not being loaded automatically as a dependency. I solved it by autoloading the module on system boot. So this is not a libvirt issue, but just for a reference, there is a BZ on this [1]. 2017-03-09 19:04:57.213+: 2003: info : libvirt version: 3.1.0 2017-03-09 19:04:57.213+: 2003: info : hostname: z-nuc-11.maas 2017-03-09 19:04:57.213+: 2003: error : virNetSocketReadWire:1800 : End of file while reading data: Input/output error I suppose this corresponds to the problem above, do you hit this error if you work around the vfio-mdev module problem described
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On 03/17/2017 01:57 AM, Chen, Xiaoguang wrote: > > >> -Original Message- >> From: sendmail [mailto:justsendmailnothinge...@gmail.com] On Behalf Of Laine >> Stump >> Sent: Thursday, March 16, 2017 10:01 PM >> To: libvir-list@redhat.com >> Cc: Chen, Xiaoguang <xiaoguang.c...@intel.com>; Erik Skultety >> <eskul...@redhat.com>; He, Yongli <yongli...@intel.com> >> Subject: Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev >> framework to libvirt >> >> On 03/16/2017 03:17 AM, Chen, Xiaoguang wrote: >>> the screen call trace while start the VM (same for Ubuntu, Win10 etc) >>> == >>> >>> ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu >>> 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 >>> 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas >>> 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : >>> Domain id=1 name='vgpu-ubuntu' >>> uuid=972b5e38-0437-11e7-8f97-d36dba74552d >>> is tainted: high-privileges >> >> I haven't considered any of the rest of the log yet, but this caught my eye >> on a >> first pass - "high-privileges" means that you're running qemu as root, so >> your test >> is bypassing several issues that could cause vfio device assignment to fail >> on a >> "standard" system. > What do you mean for 'cause vfio device assignment to fail on a standard > system'? I mean that there are some device/file setup operations that libvirt should be performing in order to make vfio device assignment work properly in the case that the qemu process is unprivileged, and those are often the source of bugs; if you run your tests with a privileged qemu process, you're not testing any of the code that performs those operations. > >> It shouldn't be necessary to run qemu as root in order for >> device assignment to work. Is there some specific reason that you're doing >> it this >> way? (I'm guessing that you've set "user = root" in /etc/libvirt/qemu.conf) > No. we will test the v3 using a non-root user. > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
>-Original Message- >From: Erik Skultety [mailto:eskul...@redhat.com] >Sent: Thursday, March 16, 2017 10:41 PM >To: Chen, Xiaoguang <xiaoguang.c...@intel.com> >Cc: libvir-list@redhat.com; He, Yongli <yongli...@intel.com> >Subject: Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev >framework to libvirt > >> [2] 2005 >> ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ >> *** >> start libvirt-d >> 2017-03-09 19:04:57.211+: 2059: info : libvirt version: 3.1.0 >> 2017-03-09 19:04:57.211+: 2059: info : hostname: z-nuc-11.maas >> 2017-03-09 19:04:57.211+: 2059: error : qemuMonitorOpenUnix:367 : >> failed to connect to monitor socket: No such process >> 2017-03-09 19:04:57.213+: 2059: error : >> virMediatedDeviceGetIOMMUGroupDev:153 : internal error: IOMMU group >> file >> /sys/bus/mdev/devices/894f3983-1a36-42b3-b52c- >1024aca216be/iommu_group >> is not a symlink > >When I saw this error message for the first time in the original thread, I got >confused, since this just checks whether the symlink exists, if it doesn't, >the vfio >device probably also doesn't exist (but take this with a grain of salt, I >haven't >investigated that deep) and libvirt needs it to pass it onto qemu command >line. I >hit this issue once by accident in the past and at that time I didn't >understand >what caused it, but after a reboot it was gone. >So seeing it here it caught my eye and I investigated it last week. What I >found >out was that it's caused by the vfio-mdev module not being loaded automatically >as a dependency. I solved it by autoloading the module on system boot. So this >is >not a libvirt issue, but just for a reference, there is a BZ on this [1]. > >> 2017-03-09 19:04:57.213+: 2003: info : libvirt version: 3.1.0 >> 2017-03-09 19:04:57.213+: 2003: info : hostname: z-nuc-11.maas >> 2017-03-09 19:04:57.213+: 2003: error : virNetSocketReadWire:1800 : >> End of file while reading data: Input/output error > >I suppose this corresponds to the problem above, do you hit this error if you >work >around the vfio-mdev module problem described above? Based on the KVMGT setup guide we should add kvmgt, vfio-iommu-type1 and vfio-mdev modules into initfamfs. So I don't think it is the same problem. But we will double confirm it later. Yong li is on vacation this week when he come back we will do the test again. > >> >> the screen call trace while start the VM (same for Ubuntu, Win10 etc) >> == >> >> ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu >> 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 >> 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas >> 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : >> Domain id=1 name='vgpu-ubuntu' >> uuid=972b5e38-0437-11e7-8f97-d36dba74552d >> is tainted: high-privileges >> 2017-03-09 19:06:50.819+: 2204: info : libvirt version: 3.1.0 >> 2017-03-09 19:06:50.819+: 2232: warning : >> virDomainAuditHostdev:456 >> : Unexpected hostdev type while encoding audit message: 4 > >This one's interesting, again, are you able to hit the error when you work >around >the missing vfio-mdev module? I'll have a look at this if you actually can hit >the >error, even if the XML is correct. > >I posted v3 of the series and also created a new branch 'mdev-next' on my >github >[2]. I dropped the attribute 'type' from the source address element, so follow >the >example in the updated docs. Will test the v3 when yong li back from vacation. > >Thanks for giving it a try. >Erik > >[1] https://bugzilla.redhat.com/show_bug.cgi?id=1420572 >[2] https://github.com/eskultety/libvirt/commits/mdev-next -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
>-Original Message- >From: sendmail [mailto:justsendmailnothinge...@gmail.com] On Behalf Of Laine >Stump >Sent: Thursday, March 16, 2017 10:01 PM >To: libvir-list@redhat.com >Cc: Chen, Xiaoguang <xiaoguang.c...@intel.com>; Erik Skultety ><eskul...@redhat.com>; He, Yongli <yongli...@intel.com> >Subject: Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev >framework to libvirt > >On 03/16/2017 03:17 AM, Chen, Xiaoguang wrote: >> the screen call trace while start the VM (same for Ubuntu, Win10 etc) >> == >> >> ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu >> 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 >> 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas >> 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : >> Domain id=1 name='vgpu-ubuntu' >> uuid=972b5e38-0437-11e7-8f97-d36dba74552d >> is tainted: high-privileges > >I haven't considered any of the rest of the log yet, but this caught my eye on >a >first pass - "high-privileges" means that you're running qemu as root, so your >test >is bypassing several issues that could cause vfio device assignment to fail on >a >"standard" system. What do you mean for 'cause vfio device assignment to fail on a standard system'? > It shouldn't be necessary to run qemu as root in order for >device assignment to work. Is there some specific reason that you're doing it >this >way? (I'm guessing that you've set "user = root" in /etc/libvirt/qemu.conf) No. we will test the v3 using a non-root user. -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
> [2] 2005 > ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ > *** > start libvirt-d > 2017-03-09 19:04:57.211+: 2059: info : libvirt version: 3.1.0 > 2017-03-09 19:04:57.211+: 2059: info : hostname: z-nuc-11.maas > 2017-03-09 19:04:57.211+: 2059: error : qemuMonitorOpenUnix:367 : > failed to connect to monitor socket: No such process > 2017-03-09 19:04:57.213+: 2059: error : > virMediatedDeviceGetIOMMUGroupDev:153 : internal error: IOMMU group file > /sys/bus/mdev/devices/894f3983-1a36-42b3-b52c-1024aca216be/iommu_group > is not a symlink When I saw this error message for the first time in the original thread, I got confused, since this just checks whether the symlink exists, if it doesn't, the vfio device probably also doesn't exist (but take this with a grain of salt, I haven't investigated that deep) and libvirt needs it to pass it onto qemu command line. I hit this issue once by accident in the past and at that time I didn't understand what caused it, but after a reboot it was gone. So seeing it here it caught my eye and I investigated it last week. What I found out was that it's caused by the vfio-mdev module not being loaded automatically as a dependency. I solved it by autoloading the module on system boot. So this is not a libvirt issue, but just for a reference, there is a BZ on this [1]. > 2017-03-09 19:04:57.213+: 2003: info : libvirt version: 3.1.0 > 2017-03-09 19:04:57.213+: 2003: info : hostname: z-nuc-11.maas > 2017-03-09 19:04:57.213+: 2003: error : virNetSocketReadWire:1800 : > End of file while reading data: Input/output error I suppose this corresponds to the problem above, do you hit this error if you work around the vfio-mdev module problem described above? > > the screen call trace while start the VM (same for Ubuntu, Win10 etc) > == > > ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu > 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 > 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas > 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : > Domain id=1 name='vgpu-ubuntu' uuid=972b5e38-0437-11e7-8f97-d36dba74552d > is tainted: high-privileges > 2017-03-09 19:06:50.819+: 2204: info : libvirt version: 3.1.0 > 2017-03-09 19:06:50.819+: 2232: warning : virDomainAuditHostdev:456 > : Unexpected hostdev type while encoding audit message: 4 This one's interesting, again, are you able to hit the error when you work around the missing vfio-mdev module? I'll have a look at this if you actually can hit the error, even if the XML is correct. I posted v3 of the series and also created a new branch 'mdev-next' on my github [2]. I dropped the attribute 'type' from the source address element, so follow the example in the updated docs. Thanks for giving it a try. Erik [1] https://bugzilla.redhat.com/show_bug.cgi?id=1420572 [2] https://github.com/eskultety/libvirt/commits/mdev-next -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On 03/16/2017 03:17 AM, Chen, Xiaoguang wrote: > the screen call trace while start the VM (same for Ubuntu, Win10 etc) > == > > ubuntu@z-nuc-11:~/vgpu-meta/libvirt-stage$ myvirsh start vgpu-ubuntu > 2017-03-09 19:06:50.483+: 2232: info : libvirt version: 3.1.0 > 2017-03-09 19:06:50.483+: 2232: info : hostname: z-nuc-11.maas > 2017-03-09 19:06:50.483+: 2232: warning : qemuDomainObjTaint:4056 : > Domain id=1 name='vgpu-ubuntu' uuid=972b5e38-0437-11e7-8f97-d36dba74552d > is tainted: high-privileges I haven't considered any of the rest of the log yet, but this caught my eye on a first pass - "high-privileges" means that you're running qemu as root, so your test is bypassing several issues that could cause vfio device assignment to fail on a "standard" system. It shouldn't be necessary to run qemu as root in order for device assignment to work. Is there some specific reason that you're doing it this way? (I'm guessing that you've set "user = root" in /etc/libvirt/qemu.conf) -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
ty >Sent: Monday, February 20, 2017 10:28 PM >To: libvir-list@redhat.com >Cc: Erik Skultety <eskul...@redhat.com> >Subject: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework >to libvirt > >since the original v2 [1]: >- resolved a few merge conflicts caused by @9d92f533 which refactored out >some duplicate code which eventually lead to dropping patch 14/18 from the >original series due to being unnecessary >- rebased onto fresh HEAD > >[1] https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html > >Erik Skultety (18): > util: Introduce new module virmdev > conf: Introduce new hostdev device type mdev > conf: Introduce new address type mdev > conf: Update XML parser, formatter, and RNG schema to support mdev > conf: Introduce virDomainHostdevDefPostParse > conf: Add post parse code for mdevs to virDomainHostdevDefPostParse > security: dac: Enable labeling of vfio mediated devices > security: selinux: Enable labeling of vfio mediated devices > conf: Enable cold-plug of a mediated device > qemu: Assign PCI addresses for mediated devices as well > hostdev: Maintain a driver list of active mediated devices > hostdev: Introduce a reattach method for mediated devices > qemu: cgroup: Adjust cgroups' logic to allow mediated devices > qemu: Bump the memory locking limit for mdevs as well > qemu: Format mdevs on qemu command line > test: Add some test cases for our test suite regarding the mdevs > docs: Document the new hostdev and address type 'mdev' > news: Update the NEWS.xml about the new mdev feature > > docs/formatdomain.html.in | 48 ++- > docs/news.xml | 9 + > docs/schemas/domaincommon.rng | 26 ++ > po/POTFILES.in | 1 + > src/Makefile.am| 1 + > src/conf/device_conf.h | 1 + > src/conf/domain_conf.c | 203 ++-- > src/conf/domain_conf.h | 9 + > src/libvirt_private.syms | 20 ++ > src/qemu/qemu_command.c| 49 +++ > src/qemu/qemu_command.h| 5 + > src/qemu/qemu_domain.c | 23 +- > src/qemu/qemu_domain.h | 1 + > src/qemu/qemu_domain_address.c | 16 +- > src/qemu/qemu_hostdev.c| 37 +++ > src/qemu/qemu_hostdev.h| 8 + > src/qemu/qemu_hotplug.c| 2 + > src/security/security_apparmor.c | 3 + > src/security/security_dac.c| 55 > src/security/security_selinux.c| 54 > src/util/virhostdev.c | 229 - > src/util/virhostdev.h | 16 + > src/util/virmdev.c | 358 + > src/util/virmdev.h | 93 ++ > tests/domaincapsschemadata/full.xml| 1 + > .../qemuxml2argv-hostdev-mdev-unmanaged.args | 25 ++ > .../qemuxml2argv-hostdev-mdev-unmanaged.xml| 37 +++ > tests/qemuxml2argvtest.c | 6 + > .../qemuxml2xmlout-hostdev-mdev-unmanaged.xml | 40 +++ > tests/qemuxml2xmltest.c| 1 + > 30 files changed, 1333 insertions(+), 44 deletions(-) create mode 100644 >src/util/virmdev.c create mode 100644 src/util/virmdev.h create mode 100644 >tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.args > create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev- >unmanaged.xml > create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-hostdev- >mdev-unmanaged.xml > >-- >2.10.2 > >-- >libvir-list mailing list >libvir-list@redhat.com >https://www.redhat.com/mailman/listinfo/libvir-list gdb-libvirt Description: gdb-libvirt myvirsh Description: myvirsh vgpu-fedrea 69dcd4dd-3f83-46d3-aa89-905e755bdb49 1953125 1953125 2 hvm destroy restart destroy /usr/bin/qemu-system-x86_64 vgpu-ubuntu 972b5e38-0437-11e7-8f97-d36dba74552d 1953125 1953125 2 hvm destroy restart destroy /usr/bin/qemu-system-x86_64
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On Mon, 27 Feb 2017 10:55:26 +0100 Erik Skultetywrote: > On Fri, Feb 24, 2017 at 11:10:00AM -0700, Alex Williamson wrote: > > On Mon, 20 Feb 2017 15:28:13 +0100 > > Erik Skultety wrote: > > > > > since the original v2 [1]: > > > - resolved a few merge conflicts caused by @9d92f533 which refactored out > > > some > > > duplicate code which eventually lead to dropping patch 14/18 from the > > > original > > > series due to being unnecessary > > > - rebased onto fresh HEAD > > > > > > [1] > > > https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html > > > > > > Erik Skultety (18): > > > util: Introduce new module virmdev > > > conf: Introduce new hostdev device type mdev > > > conf: Introduce new address type mdev > > > conf: Update XML parser, formatter, and RNG schema to support mdev > > > conf: Introduce virDomainHostdevDefPostParse > > > conf: Add post parse code for mdevs to virDomainHostdevDefPostParse > > > security: dac: Enable labeling of vfio mediated devices > > > security: selinux: Enable labeling of vfio mediated devices > > > conf: Enable cold-plug of a mediated device > > > qemu: Assign PCI addresses for mediated devices as well > > > hostdev: Maintain a driver list of active mediated devices > > > hostdev: Introduce a reattach method for mediated devices > > > qemu: cgroup: Adjust cgroups' logic to allow mediated devices > > > qemu: Bump the memory locking limit for mdevs as well > > > qemu: Format mdevs on qemu command line > > > test: Add some test cases for our test suite regarding the mdevs > > > docs: Document the new hostdev and address type 'mdev' > > > news: Update the NEWS.xml about the new mdev feature > > > > > > docs/formatdomain.html.in | 48 ++- > > > docs/news.xml | 9 + > > > docs/schemas/domaincommon.rng | 26 ++ > > > po/POTFILES.in | 1 + > > > src/Makefile.am| 1 + > > > src/conf/device_conf.h | 1 + > > > src/conf/domain_conf.c | 203 ++-- > > > src/conf/domain_conf.h | 9 + > > > src/libvirt_private.syms | 20 ++ > > > > I don't understand how these get generated, so I won't suggest where > > they should be added, but a usb test fails for me without adding > > these to this syms file: > > > > Hmm, weird, nothing fails for me, even rebased onto the current master. > Anyhow, I checked that we indeed put these generated methods into the .syms > file which I didn't know (never had a problem with that), but I'll fix it. The error occurs for me on the make rpm target with the following: FAIL: virusbtest /home/alwillia/rpmbuild/BUILD/libvirt-3.1.0/tests/.libs/lt-virusbtest: symbol lookup error: /home/alwillia/rpmbuild/BUILD/libvirt-3.1.0/tests/.libs/virusbmock.so: undefined symbol: virMediatedDeviceModelTypeFromString FAIL virusbtest (exit status: 127) I'm based on the v3.1.0-rc1 tag plus this series. Thanks, Alex -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On Fri, Feb 24, 2017 at 11:10:00AM -0700, Alex Williamson wrote: > On Mon, 20 Feb 2017 15:28:13 +0100 > Erik Skultetywrote: > > > since the original v2 [1]: > > - resolved a few merge conflicts caused by @9d92f533 which refactored out > > some > > duplicate code which eventually lead to dropping patch 14/18 from the > > original > > series due to being unnecessary > > - rebased onto fresh HEAD > > > > [1] https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html > > > > Erik Skultety (18): > > util: Introduce new module virmdev > > conf: Introduce new hostdev device type mdev > > conf: Introduce new address type mdev > > conf: Update XML parser, formatter, and RNG schema to support mdev > > conf: Introduce virDomainHostdevDefPostParse > > conf: Add post parse code for mdevs to virDomainHostdevDefPostParse > > security: dac: Enable labeling of vfio mediated devices > > security: selinux: Enable labeling of vfio mediated devices > > conf: Enable cold-plug of a mediated device > > qemu: Assign PCI addresses for mediated devices as well > > hostdev: Maintain a driver list of active mediated devices > > hostdev: Introduce a reattach method for mediated devices > > qemu: cgroup: Adjust cgroups' logic to allow mediated devices > > qemu: Bump the memory locking limit for mdevs as well > > qemu: Format mdevs on qemu command line > > test: Add some test cases for our test suite regarding the mdevs > > docs: Document the new hostdev and address type 'mdev' > > news: Update the NEWS.xml about the new mdev feature > > > > docs/formatdomain.html.in | 48 ++- > > docs/news.xml | 9 + > > docs/schemas/domaincommon.rng | 26 ++ > > po/POTFILES.in | 1 + > > src/Makefile.am| 1 + > > src/conf/device_conf.h | 1 + > > src/conf/domain_conf.c | 203 ++-- > > src/conf/domain_conf.h | 9 + > > src/libvirt_private.syms | 20 ++ > > I don't understand how these get generated, so I won't suggest where > they should be added, but a usb test fails for me without adding > these to this syms file: > Hmm, weird, nothing fails for me, even rebased onto the current master. Anyhow, I checked that we indeed put these generated methods into the .syms file which I didn't know (never had a problem with that), but I'll fix it. Thanks, Erik -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
Re: [libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
On Mon, 20 Feb 2017 15:28:13 +0100 Erik Skultetywrote: > since the original v2 [1]: > - resolved a few merge conflicts caused by @9d92f533 which refactored out some > duplicate code which eventually lead to dropping patch 14/18 from the original > series due to being unnecessary > - rebased onto fresh HEAD > > [1] https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html > > Erik Skultety (18): > util: Introduce new module virmdev > conf: Introduce new hostdev device type mdev > conf: Introduce new address type mdev > conf: Update XML parser, formatter, and RNG schema to support mdev > conf: Introduce virDomainHostdevDefPostParse > conf: Add post parse code for mdevs to virDomainHostdevDefPostParse > security: dac: Enable labeling of vfio mediated devices > security: selinux: Enable labeling of vfio mediated devices > conf: Enable cold-plug of a mediated device > qemu: Assign PCI addresses for mediated devices as well > hostdev: Maintain a driver list of active mediated devices > hostdev: Introduce a reattach method for mediated devices > qemu: cgroup: Adjust cgroups' logic to allow mediated devices > qemu: Bump the memory locking limit for mdevs as well > qemu: Format mdevs on qemu command line > test: Add some test cases for our test suite regarding the mdevs > docs: Document the new hostdev and address type 'mdev' > news: Update the NEWS.xml about the new mdev feature > > docs/formatdomain.html.in | 48 ++- > docs/news.xml | 9 + > docs/schemas/domaincommon.rng | 26 ++ > po/POTFILES.in | 1 + > src/Makefile.am| 1 + > src/conf/device_conf.h | 1 + > src/conf/domain_conf.c | 203 ++-- > src/conf/domain_conf.h | 9 + > src/libvirt_private.syms | 20 ++ I don't understand how these get generated, so I won't suggest where they should be added, but a usb test fails for me without adding these to this syms file: +virMediatedDeviceModelTypeFromString; +virMediatedDeviceModelTypeToString; Thanks, Alex > src/qemu/qemu_command.c| 49 +++ > src/qemu/qemu_command.h| 5 + > src/qemu/qemu_domain.c | 23 +- > src/qemu/qemu_domain.h | 1 + > src/qemu/qemu_domain_address.c | 16 +- > src/qemu/qemu_hostdev.c| 37 +++ > src/qemu/qemu_hostdev.h| 8 + > src/qemu/qemu_hotplug.c| 2 + > src/security/security_apparmor.c | 3 + > src/security/security_dac.c| 55 > src/security/security_selinux.c| 54 > src/util/virhostdev.c | 229 - > src/util/virhostdev.h | 16 + > src/util/virmdev.c | 358 > + > src/util/virmdev.h | 93 ++ > tests/domaincapsschemadata/full.xml| 1 + > .../qemuxml2argv-hostdev-mdev-unmanaged.args | 25 ++ > .../qemuxml2argv-hostdev-mdev-unmanaged.xml| 37 +++ > tests/qemuxml2argvtest.c | 6 + > .../qemuxml2xmlout-hostdev-mdev-unmanaged.xml | 40 +++ > tests/qemuxml2xmltest.c| 1 + > 30 files changed, 1333 insertions(+), 44 deletions(-) > create mode 100644 src/util/virmdev.c > create mode 100644 src/util/virmdev.h > create mode 100644 > tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.args > create mode 100644 > tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.xml > create mode 100644 > tests/qemuxml2xmloutdata/qemuxml2xmlout-hostdev-mdev-unmanaged.xml > -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list
[libvirt] [RFC PATCH v2 REBASE 00/18] Introduce vGPU mdev framework to libvirt
since the original v2 [1]: - resolved a few merge conflicts caused by @9d92f533 which refactored out some duplicate code which eventually lead to dropping patch 14/18 from the original series due to being unnecessary - rebased onto fresh HEAD [1] https://www.redhat.com/archives/libvir-list/2017-February/msg00739.html Erik Skultety (18): util: Introduce new module virmdev conf: Introduce new hostdev device type mdev conf: Introduce new address type mdev conf: Update XML parser, formatter, and RNG schema to support mdev conf: Introduce virDomainHostdevDefPostParse conf: Add post parse code for mdevs to virDomainHostdevDefPostParse security: dac: Enable labeling of vfio mediated devices security: selinux: Enable labeling of vfio mediated devices conf: Enable cold-plug of a mediated device qemu: Assign PCI addresses for mediated devices as well hostdev: Maintain a driver list of active mediated devices hostdev: Introduce a reattach method for mediated devices qemu: cgroup: Adjust cgroups' logic to allow mediated devices qemu: Bump the memory locking limit for mdevs as well qemu: Format mdevs on qemu command line test: Add some test cases for our test suite regarding the mdevs docs: Document the new hostdev and address type 'mdev' news: Update the NEWS.xml about the new mdev feature docs/formatdomain.html.in | 48 ++- docs/news.xml | 9 + docs/schemas/domaincommon.rng | 26 ++ po/POTFILES.in | 1 + src/Makefile.am| 1 + src/conf/device_conf.h | 1 + src/conf/domain_conf.c | 203 ++-- src/conf/domain_conf.h | 9 + src/libvirt_private.syms | 20 ++ src/qemu/qemu_command.c| 49 +++ src/qemu/qemu_command.h| 5 + src/qemu/qemu_domain.c | 23 +- src/qemu/qemu_domain.h | 1 + src/qemu/qemu_domain_address.c | 16 +- src/qemu/qemu_hostdev.c| 37 +++ src/qemu/qemu_hostdev.h| 8 + src/qemu/qemu_hotplug.c| 2 + src/security/security_apparmor.c | 3 + src/security/security_dac.c| 55 src/security/security_selinux.c| 54 src/util/virhostdev.c | 229 - src/util/virhostdev.h | 16 + src/util/virmdev.c | 358 + src/util/virmdev.h | 93 ++ tests/domaincapsschemadata/full.xml| 1 + .../qemuxml2argv-hostdev-mdev-unmanaged.args | 25 ++ .../qemuxml2argv-hostdev-mdev-unmanaged.xml| 37 +++ tests/qemuxml2argvtest.c | 6 + .../qemuxml2xmlout-hostdev-mdev-unmanaged.xml | 40 +++ tests/qemuxml2xmltest.c| 1 + 30 files changed, 1333 insertions(+), 44 deletions(-) create mode 100644 src/util/virmdev.c create mode 100644 src/util/virmdev.h create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.args create mode 100644 tests/qemuxml2argvdata/qemuxml2argv-hostdev-mdev-unmanaged.xml create mode 100644 tests/qemuxml2xmloutdata/qemuxml2xmlout-hostdev-mdev-unmanaged.xml -- 2.10.2 -- libvir-list mailing list libvir-list@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list