Re: [Qemu-devel] Assigning network devices to nested VMs results in driver errors in nested VMs
On Tue, Feb 13, 2018 at 11:44 PM, Jintack Limwrote: > Hi, > > I'm trying to assign network devices to nested VMs on x86 using KVM, > but I got network device driver errors in the nested VMs. (I've tried > this about an year ago when vIOMMU patches were not upstreamed, and I > got similar errors at that time.) > > This could be network driver issues, but I'd like to get some help if > somebody encountered similar issues. > > I'm using v4.15.0 kernel and v2.11.0 QEMU, and I followed this [1] > guide. I had no problem with assigning devices to the first level VMs > (L1 VMs). And I also checked that the devices were assigned to nested > VMs with the lspci command in the nested VMs. But network device > drivers failed to initialize the device. I tried two network cards - > Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection and > Mellanox Technologies MT27500 Family. > > Intel driver error in the nested VM looks like this. > [1.939552] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - > version 5.1.0-k > [1.949796] ixgbe: Copyright (c) 1999-2016 Intel Corporation. > [2.210024] ixgbe :00:04.0: HW Init failed: -12 > [2.218144] ixgbe: probe of :00:04.0 failed with error -12 > I was assigning PF to the L1 VMs and L2 VMs so far; I guess this is not the right way. So I tried to assign VF to the L1 VM and assigned the same VF to the L2 VM in turn. Then the device driver in L2 VM didn't show any error, and I was able to configure the network interface. But the network still didn't work. I only tried Intel network device so far. > and I saw lots of these messages in the host (L0) kernel log when > booting the nested VM. > > [ 1557.404173] DMAR: DRHD: handling fault status reg 102 > [ 1557.409813] DMAR: [DMA Read] Request device [06:00.0] fault addr > 9 [fault reason 06] PTE Read access is not set > [ 1561.383957] DMAR: DRHD: handling fault status reg 202 > [ 1561.389598] DMAR: [DMA Read] Request device [06:00.0] fault addr > 9 [fault reason 06] PTE Read access is not set > I still see similar error logs in the host kernel. The fault address looks different, though. [ 3228.636485] ixgbe :06:00.0 eth2: VF Reset msg received from vf 0 [ 3236.023683] DMAR: DRHD: handling fault status reg 2 [ 3236.029129] DMAR: [DMA Read] Request device [06:10.0] fault addr 354748000 [fault reason 06] PTE Read access is not set [ 3236.371711] DMAR: DRHD: handling fault status reg 102 [ 3236.377353] DMAR: [DMA Read] Request device [06:10.0] fault addr 354748000 [fault reason 06] PTE Read access is not set [ 3236.595667] DMAR: DRHD: handling fault status reg 202 [ 3236.601307] DMAR: [DMA Read] Request device [06:10.0] fault addr 354748000 [fault reason 06] PTE Read access is not set [ 3236.831863] DMAR: DRHD: handling fault status reg 302 [ 3236.837503] DMAR: [DMA Read] Request device [06:10.0] fault addr 370b7c000 [fault reason 06] PTE Read access is not set [ 3237.647806] vfio-pci :06:10.0: timed out waiting for pending transaction; performing function level reset anyway > This is Mellanox driver error in another nested VM. > [2.481694] mlx4_core: Initializing :00:04.0 > [3.519422] mlx4_core :00:04.0: Installed FW has unsupported > command interface revision 0 > [3.537769] mlx4_core :00:04.0: (Installed FW version is 0.0.000) > [3.551733] mlx4_core :00:04.0: This driver version supports > only revisions 2 to 3 > [3.568758] mlx4_core :00:04.0: QUERY_FW command failed, aborting > [3.582789] mlx4_core :00:04.0: Failed to init fw, aborting. > > The host showed similar messages as above. > > I wonder what could be the cause of these errors. Please let me know > if further information is needed. > > [1] https://wiki.qemu.org/Features/VT-d > > Thanks, > Jintack
Re: [Qemu-devel] Assigning network devices to nested VMs results in driver errors in nested VMs
On Wed, Feb 14, 2018 at 12:36 AM, Peter Xuwrote: > On Tue, Feb 13, 2018 at 11:44:09PM -0500, Jintack Lim wrote: >> Hi, >> >> I'm trying to assign network devices to nested VMs on x86 using KVM, >> but I got network device driver errors in the nested VMs. (I've tried >> this about an year ago when vIOMMU patches were not upstreamed, and I >> got similar errors at that time.) >> >> This could be network driver issues, but I'd like to get some help if >> somebody encountered similar issues. >> >> I'm using v4.15.0 kernel and v2.11.0 QEMU, and I followed this [1] >> guide. I had no problem with assigning devices to the first level VMs >> (L1 VMs). And I also checked that the devices were assigned to nested >> VMs with the lspci command in the nested VMs. But network device >> drivers failed to initialize the device. I tried two network cards - >> Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection and >> Mellanox Technologies MT27500 Family. >> >> Intel driver error in the nested VM looks like this. >> [1.939552] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - >> version 5.1.0-k >> [1.949796] ixgbe: Copyright (c) 1999-2016 Intel Corporation. >> [2.210024] ixgbe :00:04.0: HW Init failed: -12 >> [2.218144] ixgbe: probe of :00:04.0 failed with error -12 >> >> and I saw lots of these messages in the host (L0) kernel log when >> booting the nested VM. >> >> [ 1557.404173] DMAR: DRHD: handling fault status reg 102 >> [ 1557.409813] DMAR: [DMA Read] Request device [06:00.0] fault addr >> 9 [fault reason 06] PTE Read access is not set >> [ 1561.383957] DMAR: DRHD: handling fault status reg 202 >> [ 1561.389598] DMAR: [DMA Read] Request device [06:00.0] fault addr >> 9 [fault reason 06] PTE Read access is not set >> >> This is Mellanox driver error in another nested VM. >> [2.481694] mlx4_core: Initializing :00:04.0 >> [3.519422] mlx4_core :00:04.0: Installed FW has unsupported >> command interface revision 0 >> [3.537769] mlx4_core :00:04.0: (Installed FW version is 0.0.000) >> [3.551733] mlx4_core :00:04.0: This driver version supports >> only revisions 2 to 3 >> [3.568758] mlx4_core :00:04.0: QUERY_FW command failed, aborting >> [3.582789] mlx4_core :00:04.0: Failed to init fw, aborting. >> >> The host showed similar messages as above. >> >> I wonder what could be the cause of these errors. Please let me know >> if further information is needed. >> >> [1] https://wiki.qemu.org/Features/VT-d > > Hi, Jintack, Hi Peter, > > Thanks for reporting the problem. > > I haven't been playing with nested assignment much recently (and even > before), but I think I encountered similar problem too in the past. Oh, that's good to hear that :) > > Will let you know if I had any progress, but it's possibly not gonna > happen in a few days since there'll be a whole week holiday starting > from tomorrow (which is Chinese Spring Festival). Thanks a lot. Enjoy Lunar New Year! > > -- > Peter Xu >
Re: [Qemu-devel] Assigning network devices to nested VMs results in driver errors in nested VMs
On Tue, Feb 13, 2018 at 11:44:09PM -0500, Jintack Lim wrote: > Hi, > > I'm trying to assign network devices to nested VMs on x86 using KVM, > but I got network device driver errors in the nested VMs. (I've tried > this about an year ago when vIOMMU patches were not upstreamed, and I > got similar errors at that time.) > > This could be network driver issues, but I'd like to get some help if > somebody encountered similar issues. > > I'm using v4.15.0 kernel and v2.11.0 QEMU, and I followed this [1] > guide. I had no problem with assigning devices to the first level VMs > (L1 VMs). And I also checked that the devices were assigned to nested > VMs with the lspci command in the nested VMs. But network device > drivers failed to initialize the device. I tried two network cards - > Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection and > Mellanox Technologies MT27500 Family. > > Intel driver error in the nested VM looks like this. > [1.939552] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - > version 5.1.0-k > [1.949796] ixgbe: Copyright (c) 1999-2016 Intel Corporation. > [2.210024] ixgbe :00:04.0: HW Init failed: -12 > [2.218144] ixgbe: probe of :00:04.0 failed with error -12 > > and I saw lots of these messages in the host (L0) kernel log when > booting the nested VM. > > [ 1557.404173] DMAR: DRHD: handling fault status reg 102 > [ 1557.409813] DMAR: [DMA Read] Request device [06:00.0] fault addr > 9 [fault reason 06] PTE Read access is not set > [ 1561.383957] DMAR: DRHD: handling fault status reg 202 > [ 1561.389598] DMAR: [DMA Read] Request device [06:00.0] fault addr > 9 [fault reason 06] PTE Read access is not set > > This is Mellanox driver error in another nested VM. > [2.481694] mlx4_core: Initializing :00:04.0 > [3.519422] mlx4_core :00:04.0: Installed FW has unsupported > command interface revision 0 > [3.537769] mlx4_core :00:04.0: (Installed FW version is 0.0.000) > [3.551733] mlx4_core :00:04.0: This driver version supports > only revisions 2 to 3 > [3.568758] mlx4_core :00:04.0: QUERY_FW command failed, aborting > [3.582789] mlx4_core :00:04.0: Failed to init fw, aborting. > > The host showed similar messages as above. > > I wonder what could be the cause of these errors. Please let me know > if further information is needed. > > [1] https://wiki.qemu.org/Features/VT-d Hi, Jintack, Thanks for reporting the problem. I haven't been playing with nested assignment much recently (and even before), but I think I encountered similar problem too in the past. Will let you know if I had any progress, but it's possibly not gonna happen in a few days since there'll be a whole week holiday starting from tomorrow (which is Chinese Spring Festival). -- Peter Xu
[Qemu-devel] Assigning network devices to nested VMs results in driver errors in nested VMs
Hi, I'm trying to assign network devices to nested VMs on x86 using KVM, but I got network device driver errors in the nested VMs. (I've tried this about an year ago when vIOMMU patches were not upstreamed, and I got similar errors at that time.) This could be network driver issues, but I'd like to get some help if somebody encountered similar issues. I'm using v4.15.0 kernel and v2.11.0 QEMU, and I followed this [1] guide. I had no problem with assigning devices to the first level VMs (L1 VMs). And I also checked that the devices were assigned to nested VMs with the lspci command in the nested VMs. But network device drivers failed to initialize the device. I tried two network cards - Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection and Mellanox Technologies MT27500 Family. Intel driver error in the nested VM looks like this. [1.939552] ixgbe: Intel(R) 10 Gigabit PCI Express Network Driver - version 5.1.0-k [1.949796] ixgbe: Copyright (c) 1999-2016 Intel Corporation. [2.210024] ixgbe :00:04.0: HW Init failed: -12 [2.218144] ixgbe: probe of :00:04.0 failed with error -12 and I saw lots of these messages in the host (L0) kernel log when booting the nested VM. [ 1557.404173] DMAR: DRHD: handling fault status reg 102 [ 1557.409813] DMAR: [DMA Read] Request device [06:00.0] fault addr 9 [fault reason 06] PTE Read access is not set [ 1561.383957] DMAR: DRHD: handling fault status reg 202 [ 1561.389598] DMAR: [DMA Read] Request device [06:00.0] fault addr 9 [fault reason 06] PTE Read access is not set This is Mellanox driver error in another nested VM. [2.481694] mlx4_core: Initializing :00:04.0 [3.519422] mlx4_core :00:04.0: Installed FW has unsupported command interface revision 0 [3.537769] mlx4_core :00:04.0: (Installed FW version is 0.0.000) [3.551733] mlx4_core :00:04.0: This driver version supports only revisions 2 to 3 [3.568758] mlx4_core :00:04.0: QUERY_FW command failed, aborting [3.582789] mlx4_core :00:04.0: Failed to init fw, aborting. The host showed similar messages as above. I wonder what could be the cause of these errors. Please let me know if further information is needed. [1] https://wiki.qemu.org/Features/VT-d Thanks, Jintack