[Bug 1819989] Re: Add basic support to NVLink2 passthrough
Hello, I tested the kernel with the changes and it works nice! Thank you root@ubuntu:~# numactl -H available: 3 nodes (0,251-252) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 node 0 size: 392745 MB node 0 free: 390074 MB node 251 cpus: node 251 size: 32256 MB node 251 free: 32253 MB node 252 cpus: node 252 size: 32256 MB node 252 free: 32252 MB node distances: node 0 251 252 0: 10 40 40 251: 40 10 40 252: 40 40 10 root@ubuntu:~# nvidia-smi Fri Apr 12 13:53:42 2019 +-+ | NVIDIA-SMI 410.72 Driver Version: 410.72 CUDA Version: 10.0 | |---+--+--+ | GPU NamePersistence-M| Bus-IdDisp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===+==+==| | 0 Tesla V100-SXM2... On | 0001:00:02.0 Off |0 | | N/A 34CP041W / 300W | 3MiB / 32256MiB | 0% Default | +---+--+--+ | 1 Tesla V100-SXM2... On | 0001:00:08.0 Off |0 | | N/A 38CP043W / 300W | 3MiB / 32256MiB | 0% Default | +---+--+--+ +-+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=| | No running processes found | +-+ -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819989 Title: Add basic support to NVLink2 passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1819989/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819989] Re: Add basic support to NVLink2 passthrough
** Attachment added: "small script to detach devices from the same IOMMU group" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1819989/+attachment/5251785/+files/detach.sh -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819989 Title: Add basic support to NVLink2 passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1819989/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819989] Re: Add basic support to NVLink2 passthrough
SRU: [Impact] * An important feature was developed for PowerPC upstream and backported to a custom version of Ubuntu Bionic 18.04.1. The feature is known as nvlink2[1] passthrough, it allows physical GPUs to be accessed from any QEMU/KVM virtual machine. The problem happens when clients want to use that feature within their virtual machines: They will need to use a custom version, not the standard Ubuntu Bionic for PowerPC that everyone knows where it's and how to install it. We understand that it's a huge impact in the user experience, not only the extra-difficulty to find/install the correct version but users that misunderstand the need of a custom version will think that the feature is simply broken. Due to the fact that the guest part (the code that will run in the virtual machine) is a way simpler than the host part we decided to send the patches as a SRU. Fixing the user-experience problem without impacting existing use-cases. [1] https://wccftech.com/nvidia-volta-gv100-gpu-fast-pascal-gp100/ [Test Case] * In order to reproduce the issue, it's required a Power9 system with NVLink2 + NVidia GPU and the customized Ubuntu Bionic installed (kernel + qemu). * Then, create a virtual machine like: Create a disk image: $ qemu-img create sda.qcow2 -f qcow2 100G Find the devices to be attached: $ lspci | grep NVIDIA ... 0004:04:00.0 3D controller: NVIDIA Corporation GV100 [Tesla V100 SXM2] (rev a1) ... Detach all devices (including devices that belong to the same IOMMU group) to be passed to the virtual machine (script detach.sh attached): $ sudo ./detach.sh 0004:04:00.0 Run the virtual machine: $ sudo qemu-system-ppc64 -nodefaults \ -chardev stdio,id=STDIO0,signal=off,mux=on \ -device spapr-vty,id=svty0,reg=0x7110,chardev=STDIO0 \ -mon id=MON0,chardev=STDIO0,mode=readline \ -nographic -vga none -enable-kvm \ -device nec-usb-xhci,id=nec-usb-xhci0 \ -m 16384M \ -chardev socket,id=SOCKET0,server,nowait,host=localhost,port=4 \ -mon chardev=SOCKET0,mode=control \ -smp 16,threads=4 \ -netdev "user,id=USER0,hostfwd=tcp::-:22" \ -device "virtio-net-pci,id=vnet0,mac=C0:41:49:4b:00:00,netdev=USER0" \ -drive file=sda.qcow2,format=qcow2,if=none,id=drive-virtio-disk0 \ -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x3,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 \ -device "vfio-pci,id=vfio0004_04_00_0,host=0004:04:00.0" \ -device "vfio-pci,id=vfio0006_00_00_0,host=0006:00:00.0" \ -device "vfio-pci,id=vfio0006_00_00_1,host=0006:00:00.1" \ -device "vfio-pci,id=vfio0006_00_00_2,host=0006:00:00.2" \ -global spapr-pci-host-bridge.pgsz=0x10011000 \ -global spapr-pci-vfio-host-bridge.pgsz=0x10011000 \ -cdrom ubuntu-18.04.1-server-ppc64el.iso \ -machine pseries Install the system in the virtual machine and reboot. After booting in the installed virtual machine, download and install the drivers from cuda-repo-ubuntu1804-10-1-local-10.1.91-418.29_1.0-1_ppc64el.deb (NVidia website). With all nvidia drivers installed, check the result of the following commands: $ nvidia-smi on Nov 5 21:11:33 2018 +-+ | NVIDIA-SMI 410.72 Driver Version: 410.72 CUDA Version: 10.0 | |---+--+--+ | GPU NamePersistence-M| Bus-IdDisp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | |===+==+==| | 0 Tesla V100-SXM2... Off | 0001:00:00.0 Off |0 | | N/A 32CP051W / 300W | 0MiB / 32480MiB | 4% Default | +---+--+--+ ... +-+ | Processes: GPU Memory | | GPU PID Type Process name Usage | |=| | No running processes found | +-+ # numactl -H available: 2 nodes (0,255) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 node 0 size: 16332 MB node 0 free: 13781 MB node 255 cpus: node 255 size: 16128 MB node 255 free: 15190 MB node distances: node 0 255 0: 10 40 255: 40 10 [Fix] * The patchset can be found here: https://lists.ubuntu.com/archives/kernel-team/2019-March/099243.html There are 12 patches but the most important pieces are: - 8/12: https://lists.ubuntu.com/archives/kernel-team/2019-March/099250.html - 9/12: https://lists.ubuntu.com/archives/kernel-team/2019-March/099254.html NPU code already exists, so much of the work consists in update the code to NPU2 and to add the GPU memory in the numa layout. [Regression
[Bug 1819989] Re: Add basic support to NVLink2 passthrough
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819989 Title: Add basic support to NVLink2 passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1819989/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819989] Re: Add basic support to NVLink2 passthrough
The patchset is in the mailing list for review: https://lists.ubuntu.com/archives/kernel-team/2019-March/099243.html -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819989 Title: Add basic support to NVLink2 passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1819989/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1819989] [NEW] Add basic support to NVLink2 passthrough
Public bug reported: This bug exists to track the basic support to NVLink2 passthrough on Ubuntu 18.04 - for the guest side only. There's a relative small patchset that I'm going to send to Canonical Kernel Team using this buglink. On the host side we'll be running a custom version of Ubuntu 18.04 (kernel + qemu). However on the guest side it will be *very important* for clients to simply download the Ubuntu 18.04 from Canonical's website and have the NVLink2 working out of the box. For that, we have worked on a small patchset using only upstream patches without changing beyond our area. As soon as I send the patchset to the mailing list I'll update this bug with a link to that message. Thank you very much, Jose R. Ziviani ** Affects: linux (Ubuntu) Importance: Undecided Status: New ** Tags: architecture-ppc64le -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1819989 Title: Add basic support to NVLink2 passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1819989/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
Juerg, sorry, my bad. I tested my own build. I was sure I had 'apt-get' that IBM-cloud, which didn't happen. Anyway, I was not aware of that salesforce process. I saw this bug marked as "won't fix" without any further explanation, how could I know that? Anyway, how do I access the salesforce to make that request? Thanks -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
Hello Juerg, Thanks for the packages provided. I made a initial setup and it looks good. However, a bug was identified and a new patch should be applied on top of my previous patchset. Would you mind to include it? The patch is: https://github.com/jrziviani/linux-devel/commit/b466f257ada3b408be3ca475ce6253b80efc252c (also attached). Thank you very much, Jose ** Patch added: "0001-powerpc-powernv-npu-Fault-user-page-into-the-hypervi.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5231782/+files/0001-powerpc-powernv-npu-Fault-user-page-into-the-hypervi.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
Hi Christian, You're right, we're going to use our customized QEMU. Thank you!! Jose -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
Hello Christian! That's a good point. :-) As far as I know the project should be based on top of v2.11.2 (at least, this is what we've been testing). I'll try to confirm it here. Thank you -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
@Christian, hello! I completed the backport to QEMU v2.11.2, but more patches were required in order to make it work. The patchset is attached here but it can be found also at https://github.com/jrziviani/qemu-devel/tree/nv2_scratch_181204_bionic. Patches from 1 to 34 are already upstream, from 34 to 45 are under review. Thank you very much! ** Attachment added: "qemu-patches-v2.11.2.tar.gz" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5219808/+files/qemu-patches-v2.11.2.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support for NVIDIA GPU passthrough
@Juerg Hello, I have a new patchset. It has all the necessary fixes to stabilize nvlink2 passthrough, which required more patches backported. I'm attaching the tar.gz here. You can also find it at https://github.com/jrziviani/linux-devel/tree/nv2_scratch_181203_bionic It's built on top of ubuntu-bionic/master-next (Ubuntu-4.15.0-43.46). Patches from 1 to 91 are upstream, others can be found at https://github.com/aik/linux/tree/nv2. Thank you very much! ** Attachment added: "linux-patches-bionic.tar.gz" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5219785/+files/linux-patches-bionic.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support for NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support to NVIDIA GPU passthrough
The patch attached here contains the necessary to build QEMU with support to nvlink2 passthrough: 0001-qdev-Use-string-for-QOM-string-properties.patch 0002-ppc-spapr-Receive-and-store-device-tree-blob-from-SL.patch 0003-DBG-store-fdt.patch 0004-vfio-spapr-Fix-indirect-levels-calculation.patch 0005-headers-update.patch 0006-pci-Move-NVIDIA-vendor-id-to-the-rest-of-ids.patch 0007-RFC-vfio-nvidia-v100-Disable-VBIOS-update.patch 0008-spapr-iommu-Always-advertise-the-maximum-possible-DM.patch 0009-FIXME-vfio-Do-not-replay-IOMMU-mappings.patch 0010-vfio-Make-vfio_get_region_info_cap-public.patch 0011-vfio-spapr-Try-allocating-less-levels-if-failed-with.patch 0012-spapr-Add-NVLink2-memory-to-PHB-placement.patch 0013-spapr-Create-ibm-gpu-and-ibm-npu-cross-links-in-the-.patch 0014-spapr-vfio-Map-GPU-RAM-and-advertise-to-the-guest.patch 0015-debug-Disables-KVM-TCE-acceleration-and-direct-ATSD-.patch https://github.com/aik/qemu/tree/nv2 ** Attachment added: "qemu-patches.tar.gz" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5209845/+files/qemu-patches.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support to NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support to NVIDIA GPU passthrough
The file passthrough-patches.tar.gz contains the following patches: 0001-kvm-no-need-to-check-return-value-of-debugfs_create-.patch 0002-powerpc-powernv-idoa-Remove-unnecessary-pcidev-from-.patch 0003-powerpc-Use-sizeof-foo-rather-than-sizeof-struct-foo.patch 0004-powerpc-powernv-npu-Do-not-try-invalidating-32bit-ta.patch 0005-powerpc-ioda-Use-ibm-supported-tce-sizes-for-IOMMU-p.patch 0006-powerpc-io-Add-__raw_writeq_be-__raw_rm_writeq_be.patch 0007-powerpc-powernv-Use-__raw_-rm_-writeq_be-in-pci-ioda.patch 0008-powerpc-powernv-ioda2-Remove-redundant-free-of-TCE-p.patch 0009-powerpc-powernv-ioda2-Reduce-upper-limit-for-DMA-win.patch 0010-Revert-cxl-Add-kernel-API-to-allow-a-context-to-oper.patch 0011-Revert-cxl-Add-support-for-interrupts-on-the-Mellano.patch 0012-Revert-cxl-Add-cxl_check_and_switch_mode-API-to-swit.patch 0013-Revert-cxl-Add-support-for-using-the-kernel-API-with.patch 0014-Revert-powerpc-powernv-Add-support-for-the-cxl-kerne.patch 0015-Revert-cxl-Add-cxl_slot_is_supported-API.patch 0016-cxl-Remove-abandonned-capi-support-for-the-Mellanox-.patch 0017-powerpc-powernv-ioda2-Add-256M-IOMMU-page-size-to-th.patch 0018-powerpc-powernv-Remove-useless-wrapper.patch 0019-powerpc-powernv-Move-TCE-manupulation-code-to-its-ow.patch 0020-KVM-PPC-Make-iommu_table-it_userspace-big-endian.patch 0021-powerpc-powernv-Add-indirect-levels-to-it_userspace.patch 0022-powerpc-powernv-Rework-TCE-level-allocation.patch 0023-powerpc-powernv-ioda-Allocate-indirect-TCE-levels-on.patch 0024-KVM-PPC-Validate-all-tces-before-updating-tables.patch 0025-KVM-PPC-Inform-the-userspace-about-TCE-update-failur.patch 0026-KVM-PPC-Validate-TCEs-against-preregistered-memory-p.patch 0027-KVM-PPC-Avoid-marking-DMA-mapped-pages-dirty-in-real.patch 0028-KVM-PPC-Propagate-errors-to-the-guest-when-failed-in.patch 0029-KVM-PPC-Remove-redundand-permission-bits-removal.patch 0030-vfio-pci-Quiet-broken-INTx-whining-when-INTx-is-unsu.patch 0031-KVM-PPC-Book3S-HV-Add-a-debugfs-file-to-dump-radix-m.patch 0032-cxl-Remove-unused-include.patch 0033-powerpc-powernv-ioda2-Reduce-upper-limit-for-DMA-win.patch 0034-powerpc-powernv-ioda-Allocate-indirect-TCE-levels-of.patch 0035-powerpc-pseries-iommu-Allow-dynamic-window-to-start-.patch 0036-KVM-PPC-Optimize-clearing-TCEs-for-sparse-tables.patch 0037-powerpc-powernv-npu-Add-a-debugfs-setting-to-change-.patch 0038-powerpc-powernv-npu-Remove-unused-headers-and-a-macr.patch 0039-KVM-PPC-Expose-userspace-mm-context-id-via-debugfs.patch 0040-powerpc-ioda-npu2-Call-hot-reset-skiboot-hook-when-d.patch 0041-vfio-spapr_tce-Get-rid-of-possible-infinite-loop.patch 0042-vfio-spapr_tce-Simplify-page-contained-test.patch 0043-powerpc-iommu_context-Change-referencing-in-API.patch 0044-powerpc-iommu-Do-not-pin-memory-of-a-memory-device.patch 0045-vfio_pci-Allow-mapping-extra-regions.patch 0046-vfio_pci-Allow-regions-to-add-own-capabilities.patch 0047-powerpc-powernv-npu-Simplify-nestMMU-flush-flag-copy.patch 0048-powerpc-npu-dma-Add-helper-to-access-struct-npu-for-.patch 0049-powerpc-powernv-npu-Collect-all-static-symbols-under.patch 0050-FIXME-powerpc-powernv-Detach-npu-struct-from-pnv_phb.patch 0051-powerpc-pseries-iommu-Force-default-DMA-window-remov.patch 0052-powerpc-pseries-iommu-Use-memory-nodes-in-max-RAM-ad.patch 0053-FIXME-powerpc-pseries-npu2-Enable-platform-support.patch 0054-vfio_pci-Add-NVIDIA-GV100GL-Tesla-V100-SXM2-10de-1db.patch They were generated based on master-next@git://kernel.ubuntu.com/ubuntu/ubuntu-bionic.git These are related to Linux KVM only. ** Patch removed: "0001-kvm-no-need-to-check-return-value-of-debugfs_create-.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5209841/+files/0001-kvm-no-need-to-check-return-value-of-debugfs_create-.patch ** Attachment added: "patches" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5209842/+files/passthrough-patches.tar.gz -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support to NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1800649] Re: Add support to NVIDIA GPU passthrough
** Patch added: "0001-kvm-no-need-to-check-return-value-of-debugfs_create-.patch" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+attachment/5209841/+files/0001-kvm-no-need-to-check-return-value-of-debugfs_create-.patch -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1800649 Title: Add support to NVIDIA GPU passthrough To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1800649/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1686019] Re: Ubuntu 16.04.3: Qemu fails on P9
** Tags removed: verification-needed-zesty ** Tags added: verification-done-zesty -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1686019 Title: Ubuntu 16.04.3: Qemu fails on P9 To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu-power-systems/+bug/1686019/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
