Re: [vfio-users] Nested VFIO with QEMU

2019-11-05 Thread Samuel Ortiz 
On Tue, Nov 05, 2019 at 04:58:34PM -0700, Alex Williamson wrote:
> On Wed, 6 Nov 2019 00:29:52 +0100
> Samuel Ortiz  wrote:
> 
> > On Tue, Nov 05, 2019 at 01:21:48PM -0700, Alex Williamson wrote:
> > > On Fri, 18 Oct 2019 05:48:49 +
> > > "Boeuf, Sebastien"  wrote:
> > >   
> > > > Hi folks,
> > > > 
> > > > I have been recently working with VFIO, and particularly trying to
> > > > achieve device passthrough through multiple layers of virtualization.
> > > > 
> > > > I wanted to assess QEMU's performances with nested VFIO, using the
> > > > emulated Intel IOMMU device. Unfortunately, I cannot make any of my
> > > > physical device work when I pass them through, attached to the emulated
> > > > Intel IOMMU. Using regular VFIO works properly, but as soon as I enable
> > > > the virtual IOMMU, the driver fails to probe (I tried on two different
> > > > machines with different types of NIC).
> > > > 
> > > > So I was wondering if someone was aware of any issue with using both
> > > > VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be missing
> > > > something obvious but I couldn't find it so far.  
> > > 
> > > It's not something I test regularly, but I'm under the impression that
> > > nested device assignment does work.  When you say the driver fails to
> > > probe, which driver is that, the endpoint driver in the L2 guest or
> > > vfio-pci in the L1 guest?  Perhaps share your XML or command line?  
> > 
> > This is fixed now. Apparently the iommu device needs to be passed
> > _before_ the other devices on the command line. We managed to make it
> > work as expected.
> 
> Good news!
> 
> > Sebastien and Yi Liu figured this out but for some reasons the
> > thread moved to vfio-users-boun...@redhat.com.
> 
> Yes, I see some uncaught bounce notifications, it looks like Yi's
> initial reply was to vfio-users-bounces.  Yi, you might want to
> checkout your mailer configuration.  For posterity/follow-up, I'll
> paste the final message from the bounce notification below.  
Thanks for doing that.

Cheers,
Samuel.
-
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Nested VFIO with QEMU

2019-11-05 Thread Alex Williamson 
On Wed, 6 Nov 2019 00:29:52 +0100
Samuel Ortiz  wrote:

> On Tue, Nov 05, 2019 at 01:21:48PM -0700, Alex Williamson wrote:
> > On Fri, 18 Oct 2019 05:48:49 +
> > "Boeuf, Sebastien"  wrote:
> >   
> > > Hi folks,
> > > 
> > > I have been recently working with VFIO, and particularly trying to
> > > achieve device passthrough through multiple layers of virtualization.
> > > 
> > > I wanted to assess QEMU's performances with nested VFIO, using the
> > > emulated Intel IOMMU device. Unfortunately, I cannot make any of my
> > > physical device work when I pass them through, attached to the emulated
> > > Intel IOMMU. Using regular VFIO works properly, but as soon as I enable
> > > the virtual IOMMU, the driver fails to probe (I tried on two different
> > > machines with different types of NIC).
> > > 
> > > So I was wondering if someone was aware of any issue with using both
> > > VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be missing
> > > something obvious but I couldn't find it so far.  
> > 
> > It's not something I test regularly, but I'm under the impression that
> > nested device assignment does work.  When you say the driver fails to
> > probe, which driver is that, the endpoint driver in the L2 guest or
> > vfio-pci in the L1 guest?  Perhaps share your XML or command line?  
> 
> This is fixed now. Apparently the iommu device needs to be passed
> _before_ the other devices on the command line. We managed to make it
> work as expected.

Good news!

> Sebastien and Yi Liu figured this out but for some reasons the
> thread moved to vfio-users-boun...@redhat.com.

Yes, I see some uncaught bounce notifications, it looks like Yi's
initial reply was to vfio-users-bounces.  Yi, you might want to
checkout your mailer configuration.  For posterity/follow-up, I'll
paste the final message from the bounce notification below.  Thanks,

Alex

On Mon, 28 Oct 2019 08:13:23 +
"Liu, Yi L"  wrote:

> Hi Sebastien,
> 
> That’s great it works for you. I remember there was an effort
> to fix it in community. But I cannot recall if it was documented.
> If not, I think I can co-work with community to make it clear.
> 
> Regards,
> Yi Liu
> 
> From: Boeuf, Sebastien
> Sent: Friday, October 25, 2019 7:17 PM
> To: Liu, Yi L 
> Cc: Ortiz, Samuel ; vfio-users-boun...@redhat.com; 
> Bradford, Robert 
> Subject: Re: [vfio-users] Nested VFIO with QEMU
> 
> Hi Yi Liu,
> 
> Yes that was it :)
> Thank you very much for your help!
> 
> Is it documented somewhere that parameters order matters?
> 
> Thanks,
> Sebastien
> 
> On Fri, 2019-10-25 at 09:52 +0800, Liu, Yi L wrote:
> Hi Sebastien,
> 
> I guess the cmdline is cause. You should put the intel-iommu exposure prior 
> to other devices as below.
> 
> -drive if=none,id=drive0,format=raw,file=/home/sebastien/clear-kvm.img \
> -device intel-iommu,intremap=on,caching-mode=on
> -device virtio-blk-pci,drive=drive0,scsi=off \
> -device virtio-rng-pci \
> -device vfio-pci,host=00:19.0 \
> 
> Regards,
> Yi Liu
> 
> From: Boeuf, Sebastien
> Sent: Friday, October 25, 2019 7:14 AM
> To: Liu, Yi L mailto:yi.l@intel.com>>
> Cc: Ortiz, Samuel mailto:samuel.or...@intel.com>>; 
> vfio-users-boun...@redhat.com; 
> Bradford, Robert mailto:robert.bradf...@intel.com>>
> Subject: Re: [vfio-users] Nested VFIO with QEMU
> 
> Hi Yi Liu,
> 
> On Tue, 2019-10-22 at 11:01 +0800, Liu, Yi L wrote:
> 
> Hi Sebastien,
> 
> 
> 
> > From: vfio-users-boun...@redhat.com 
> > [mailto:vfio-users-  
> 
> > boun...@redhat.com] On Behalf Of Boeuf, 
> > Sebastien  
> 
> > Sent: Friday, October 18, 2019 1:49 PM  
> 
> > To: vfio-users@redhat.com  
> 
> > Cc: Ortiz, Samuel mailto:samuel.or...@intel.com>>; 
> > Bradford, Robert  
> 
> > mailto:robert.bradf...@intel.com>>  
> 
> > Subject: [vfio-users] Nested VFIO with QEMU  
> 
> >  
> 
> > Hi folks,  
> 
> >  
> 
> > I have been recently working with VFIO, and particularly trying to  
> 
> > achieve device passthrough through multiple layers of virtualization.  
> 
> >  
> 
> > I wanted to assess QEMU's performances with nested VFIO, using the  
> 
> > emulated Intel IOMMU device. Unfortunately, I cannot make any of my  
> 
> > physical device work when I pass them through, attached to the  
> 
> > emulated Intel IOMMU. Using regular VFIO works properly, but as soon  
> 
> 
> 
> Sorry, what does regular VFIO mean here?
> 
> Sorry, what I called "regular VFIO" is for the case where VFIO is not run 
> along with
> vIOMMU.
> 
> 
> 
> 
> 
> 
> > as I enable the virtual IOMMU, the driver fails to probe (I tried on  
> 
> > two different machines with different types of NIC).  
> 
> 
> 
> Ok, so regular VFIO means passthru a device to a VM which has no vIOMMU?
> 
> Yes.
> 
> 
> 
> 
> 
> 
> > So I was wondering if someone was aware of any issue with using both  
> 
> > VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be

Re: [vfio-users] Nested VFIO with QEMU

2019-11-05 Thread Samuel Ortiz 
On Tue, Nov 05, 2019 at 01:21:48PM -0700, Alex Williamson wrote:
> On Fri, 18 Oct 2019 05:48:49 +
> "Boeuf, Sebastien"  wrote:
> 
> > Hi folks,
> > 
> > I have been recently working with VFIO, and particularly trying to
> > achieve device passthrough through multiple layers of virtualization.
> > 
> > I wanted to assess QEMU's performances with nested VFIO, using the
> > emulated Intel IOMMU device. Unfortunately, I cannot make any of my
> > physical device work when I pass them through, attached to the emulated
> > Intel IOMMU. Using regular VFIO works properly, but as soon as I enable
> > the virtual IOMMU, the driver fails to probe (I tried on two different
> > machines with different types of NIC).
> > 
> > So I was wondering if someone was aware of any issue with using both
> > VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be missing
> > something obvious but I couldn't find it so far.
> 
> It's not something I test regularly, but I'm under the impression that
> nested device assignment does work.  When you say the driver fails to
> probe, which driver is that, the endpoint driver in the L2 guest or
> vfio-pci in the L1 guest?  Perhaps share your XML or command line?

This is fixed now. Apparently the iommu device needs to be passed
_before_ the other devices on the command line. We managed to make it
work as expected.
Sebastien and Yi Liu figured this out but for some reasons the
thread moved to vfio-users-boun...@redhat.com.

Cheers,
Samuel.
-
Intel Corporation SAS (French simplified joint stock company)
Registered headquarters: "Les Montalets"- 2, rue de Paris, 
92196 Meudon Cedex, France
Registration Number:  302 456 199 R.C.S. NANTERRE
Capital: 4,572,000 Euros

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Posted interrupts for nested virtualization?

2019-11-05 Thread Alex Williamson 
On Fri, 18 Oct 2019 06:08:31 +
"Boeuf, Sebastien"  wrote:

> Hi folks,
> 
> We have been recently implementing a nested VFIO solution for our Cloud
> Hypervisor VMM. Thanks to virtio-iommu, we can now pass a device
> through nested virtualization.
> 
> After some performances testing, we realized the device in L2 was
> slightly less performant than the same device in L1. The DMAR has
> nothing to do with it since it is programmed only at boot time with the
> entire guest RAM mapping for each device, but the problem comes from
> the interrupts. When using polling instead of interrupts, we can
> clearly see the device in L2 behaving very closely to what is expected
> in L1.
> 
> So basically, because the interrupt has to bounce through each layer of
> virtualization, the more layers the less performance we will get out of
> this device, right?

Yes

> We were wondering if there was any on going work to allow posted
> interrupts to be delivered directly to the last layer of virtualization
> where the device is actually used. I might be missing something but in
> theory, it seems feasible, the problem seems to be that VFIO does not
> update the IRT of the IOMMU, so I don't know how we could manually
> update it, the same way we can update the DMAR with VFIO_MAP_DMA ioctl
> for instance.
> And the second part is about knowing which interrupt needs to be
> updated in the IRT, which means the virtio-iommu device should be able
> to provide those kind of information.
> 
> It seems like there are some possibilities but I clearly don't have the
> whole picture in mind, and I'm interested in learning more about any
> existing work on this topic.

I agree with your assessment, it seems like it should be possible, but
I don't know of any work to support it.  Note that vfio doesn't
directly manipulate posted interrupts, it's done via the irqbypass
producer/consumer mechanism, so perhaps this largely needs to be done
in KVM.  I can't say I've spent many cycles thinking about it either
though.  Thanks,

Alex

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users

Re: [vfio-users] Nested VFIO with QEMU

2019-11-05 Thread Alex Williamson 
On Fri, 18 Oct 2019 05:48:49 +
"Boeuf, Sebastien"  wrote:

> Hi folks,
> 
> I have been recently working with VFIO, and particularly trying to
> achieve device passthrough through multiple layers of virtualization.
> 
> I wanted to assess QEMU's performances with nested VFIO, using the
> emulated Intel IOMMU device. Unfortunately, I cannot make any of my
> physical device work when I pass them through, attached to the emulated
> Intel IOMMU. Using regular VFIO works properly, but as soon as I enable
> the virtual IOMMU, the driver fails to probe (I tried on two different
> machines with different types of NIC).
> 
> So I was wondering if someone was aware of any issue with using both
> VFIO and virtual Intel IOMMU with QEMU? I'm sure I might be missing
> something obvious but I couldn't find it so far.

It's not something I test regularly, but I'm under the impression that
nested device assignment does work.  When you say the driver fails to
probe, which driver is that, the endpoint driver in the L2 guest or
vfio-pci in the L1 guest?  Perhaps share your XML or command line?
Thanks,

Alex

___
vfio-users mailing list
vfio-users@redhat.com
https://www.redhat.com/mailman/listinfo/vfio-users