Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Mon, 2015-06-29 at 13:27 +, Wu, Feng wrote:
>
>
> > -Original Message-
> > From: Alex Williamson [mailto:alex.william...@redhat.com]
> > Sent: Friday, June 19, 2015 4:04 AM
> > To: Wu, Feng
> > Cc: Eric Auger; Avi Kivity; kvm@vger.kernel.org;
> > linux-ker...@vger.kernel.org;
> > pbonz...@redhat.com; mtosa...@redhat.com; Joerg Roedel
> > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> >
> > [Adding Joerg since he was part of this original idea]
> >
> >
> > There are plenty of details to be filled in, but I think the basics
> > looks something like the code below. The IRQ bypass manager just
> > defines a pair of structures, one for interrupt producers and one for
> > interrupt consumers. I'm certain that we'll need more callbacks than
> > I've defined below, but figuring out what those should be for the best
> > abstraction is the hardest part of this idea. The manager provides both
> > registration and de-registration interfaces for both types of objects
> > and keeps lists for each, protected by a lock. The manager doesn't even
> > really need to know what the match token is, but I assume for our
> > purposes it will be an eventfd_ctx.
> >
> > On the vfio side, the producer struct would be embedded in the
> > vfio_pci_irq_ctx struct. KVM would probably embed the consumer struct
> > in _irqfd. As I've coded below, the IRQ bypass manager calls the
> > consumer callbacks, so the producer struct would need fields or
> > callbacks to provide the consumer the info it needs. AIUI the Posted
> > Interrupt model, VFIO only needs to provide data to the consumer. For
> > IRQ Forwarding, I think the producer needs to be informed when bypass is
> > active to model the incoming interrupt as edge vs level.
> >
> > I've prototyped the base IRQ bypass manager here as static, but I don't
> > see any reason it couldn't be a module that's loaded by dependency when
> > either vfio-pci or kvm-intel is loaded (or other producer/consumer
> > objects).
> >
> > Is this a reasonable starting point to craft the additional fields and
> > callbacks and interaction of who calls who that we need to support
> > Posted Interrupts and IRQ Forwarding? Is the AMD version of this still
> > alive? Thanks,
> >
> > Alex
> >
> > diff --git a/arch/x86/kvm/Kconfig b/arch/x86/kvm/Kconfig
> > index 413a7bf..22f6fcb 100644
> > --- a/arch/x86/kvm/Kconfig
> > +++ b/arch/x86/kvm/Kconfig
> > @@ -61,6 +61,7 @@ config KVM_INTEL
> > depends on KVM
> > # for perf_guest_get_msrs():
> > depends on CPU_SUP_INTEL
> > + select IRQ_BYPASS_MANAGER
> > ---help---
> > Provides support for KVM on Intel processors equipped with the VT
> > extensions.
> > diff --git a/drivers/vfio/pci/Kconfig b/drivers/vfio/pci/Kconfig
> > index 579d83b..02912f1 100644
> > --- a/drivers/vfio/pci/Kconfig
> > +++ b/drivers/vfio/pci/Kconfig
> > @@ -2,6 +2,7 @@ config VFIO_PCI
> > tristate "VFIO support for PCI devices"
> > depends on VFIO && PCI && EVENTFD
> > select VFIO_VIRQFD
> > + select IRQ_BYPASS_MANAGER
> > help
> > Support for the PCI VFIO bus driver. This is required to make
> > use of PCI drivers using the VFIO framework.
> > diff --git a/drivers/vfio/pci/vfio_pci_intrs.c
> > b/drivers/vfio/pci/vfio_pci_intrs.c
> > index 1f577b4..4e053be 100644
> > --- a/drivers/vfio/pci/vfio_pci_intrs.c
> > +++ b/drivers/vfio/pci/vfio_pci_intrs.c
> > @@ -181,6 +181,7 @@ static int vfio_intx_set_signal(struct vfio_pci_device
> > *vdev, int fd)
> >
> > if (vdev->ctx[0].trigger) {
> > free_irq(pdev->irq, vdev);
> > + /* irq_bypass_unregister_producer(); */
> > kfree(vdev->ctx[0].name);
> > eventfd_ctx_put(vdev->ctx[0].trigger);
> > vdev->ctx[0].trigger = NULL;
> > @@ -214,6 +215,8 @@ static int vfio_intx_set_signal(struct vfio_pci_device
> > *vdev, int fd)
> > return ret;
> > }
> >
> > + /* irq_bypass_register_producer(); */
> > +
> > /*
> > * INTx disable will stick across the new irq setup,
> > * disable_irq won't.
> > @@ -319,6 +322,7 @@ static int vfio_msi_set_vector_signal(struct
> > vfio_pci_device *vdev,
> >
> > if (vdev->ctx[vector].trigger) {
> > free_irq(irq, vdev->ctx[vector].trigger
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message- > From: Alex Williamson [mailto:alex.william...@redhat.com] > Sent: Friday, June 19, 2015 4:04 AM > To: Wu, Feng > Cc: Eric Auger; Avi Kivity; kvm@vger.kernel.org; linux-ker...@vger.kernel.org; > pbonz...@redhat.com; mtosa...@redhat.com; Joerg Roedel > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > [Adding Joerg since he was part of this original idea] > > On Thu, 2015-06-18 at 09:16 +, Wu, Feng wrote: > > > > > > > -Original Message- > > > From: Alex Williamson [mailto:alex.william...@redhat.com] > > > Sent: Tuesday, June 16, 2015 12:45 AM > > > To: Eric Auger > > > Cc: Avi Kivity; Wu, Feng; kvm@vger.kernel.org; > linux-ker...@vger.kernel.org; > > > pbonz...@redhat.com; mtosa...@redhat.com > > > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > > > > On Mon, 2015-06-15 at 18:17 +0200, Eric Auger wrote: > > > > Hi Alex, all, > > > > On 06/12/2015 09:03 PM, Alex Williamson wrote: > > > > > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: > > > > >> On 06/12/2015 06:41 PM, Alex Williamson wrote: > > > > >>> On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > > > > >>>>> -Original Message- > > > > >>>>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] > > > > >>>>> Sent: Friday, June 12, 2015 3:59 AM > > > > >>>>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > > > > >>>>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; > > > > >>>>> alex.william...@redhat.com; eric.au...@linaro.org > > > > >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > > >>>>> > > > > >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote: > > > > >>>>>> From: Eric Auger > > > > >>>>>> > > > > >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE > > > group > > > > >>>>>> and 2 device attributes: > KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > > > > >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be > > > able > > > > >>>>>> to set a VFIO device IRQ as forwarded or not forwarded. > > > > >>>>>> the command takes as argument a handle to a new struct named > > > > >>>>>> kvm_vfio_dev_irq. > > > > >>>>> Is there no way to do this automatically? After all, vfio knows > > > > >>>>> that > a > > > > >>>>> device interrupt is forwarded to some eventfd, and kvm knows that > > > some > > > > >>>>> eventfd is forwarded to a guest interrupt. If they compare notes > > > > >>>>> through a central registry, they can figure out that the interrupt > needs > > > > >>>>> to be forwarded. > > > > >>>> Oh, just like Eric mentioned in his reply, this description is out > > > > >>>> of > context > > > of > > > > >>>> this series, I will remove them in the next version. > > > > >>> > > > > >>> I suspect Avi's question was more general. While forward/unforward > is > > > > >>> out of context for this series, it's very similar in nature to > > > > >>> enabling/disabling posted interrupts. So I think the question > > > > >>> remains > > > > >>> whether we really need userspace to participate in creating this > > > > >>> shortcut or if kvm and vfio can some how orchestrate figuring it out > > > > >>> automatically. > > > > >>> > > > > >>> Personally I don't know how we could do it automatically. We've > always > > > > >>> relied on userspace to independently setup vfio and kvm such that > > > > >>> neither have any idea that the other is there and update each side > > > > >>> independently when anything changes. So it seems consistent to > > > continue > > > > >>> that here. It doesn't seem like there's much to gain > performance-wise > > > > >>> either, updates should be a relatively rare event I&
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message- > From: Joerg Roedel [mailto:j...@8bytes.org] > Sent: Monday, June 29, 2015 5:23 PM > To: Wu, Feng > Cc: Alex Williamson; Eric Auger; Avi Kivity; kvm@vger.kernel.org; > linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > On Mon, Jun 29, 2015 at 09:14:54AM +, Wu, Feng wrote: > > Do you mean updating the hardware IRTEs for all the entries in the irq > > routing table, no matter whether it is the updated one? > > Right, that's what I mean. It seems wrong to me to work around the API > interface by creating a diff between the old and the new routing table. Yes the original usage model here doesn't care about the diff between the old and new, it is a little intrusive to add the comparison code here. > It is much simpler (and easier to maintain) to just update the IRTE > and PI structures for all IRQs in the routing table, especially since > this is not a hot-path. Agree. Thanks, Feng > > > Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Mon, Jun 29, 2015 at 09:14:54AM +, Wu, Feng wrote: > Do you mean updating the hardware IRTEs for all the entries in the irq > routing table, no matter whether it is the updated one? Right, that's what I mean. It seems wrong to me to work around the API interface by creating a diff between the old and the new routing table. It is much simpler (and easier to maintain) to just update the IRTE and PI structures for all IRQs in the routing table, especially since this is not a hot-path. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message- > From: Joerg Roedel [mailto:j...@8bytes.org] > Sent: Monday, June 29, 2015 5:06 PM > To: Wu, Feng > Cc: Alex Williamson; Eric Auger; Avi Kivity; kvm@vger.kernel.org; > linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > Hi Feng, > > On Thu, Jun 25, 2015 at 09:11:52AM -0600, Alex Williamson wrote: > > So the trouble is that QEMU vfio updates a single MSI vector, but that > > just updates a single entry within a whole table of routes, then the > > whole table is pushed to KVM. But in kvm_set_irq_routing() we have > > access to both the new and the old tables, so we do have the ability to > > detect the change. We can therefore detect which GSI changed and > > cross-reference that to KVMs irqfds. If we have an irqfd that matches > > the GSI then we have all the information we need, right? We can use the > > eventfd_ctx of the irqfd to call into the IRQ bypass manager if we need > > to. If it's an irqfd that's already enabled for bypass then we may > > already have the data we need to tweak the PI config. > > > > Yes, I agree it's more difficult, but it doesn't appear to be > > impossible, right? > > Since this also doesn't happen very often, you could also just update _all_ > PI data-structures from kvm_set_irq_routing, no? This would just > resemble the way the API works anyway. Thanks a lot for your suggestion, Joerg! Do you mean updating the hardware IRTEs for all the entries in the irq routing table, no matter whether it is the updated one? Thanks, Feng > > You just need to be careful to update the data structures only when the > function can't fail anymore, so that you don't have to roll back > anything. > > > Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
Hi Feng, On Thu, Jun 25, 2015 at 09:11:52AM -0600, Alex Williamson wrote: > So the trouble is that QEMU vfio updates a single MSI vector, but that > just updates a single entry within a whole table of routes, then the > whole table is pushed to KVM. But in kvm_set_irq_routing() we have > access to both the new and the old tables, so we do have the ability to > detect the change. We can therefore detect which GSI changed and > cross-reference that to KVMs irqfds. If we have an irqfd that matches > the GSI then we have all the information we need, right? We can use the > eventfd_ctx of the irqfd to call into the IRQ bypass manager if we need > to. If it's an irqfd that's already enabled for bypass then we may > already have the data we need to tweak the PI config. > > Yes, I agree it's more difficult, but it doesn't appear to be > impossible, right? Since this also doesn't happen very often, you could also just update _all_ PI data-structures from kvm_set_irq_routing, no? This would just resemble the way the API works anyway. You just need to be careful to update the data structures only when the function can't fail anymore, so that you don't have to roll back anything. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Thu, 2015-06-25 at 09:37 +, Wu, Feng wrote: > > > -Original Message- > > From: Joerg Roedel [mailto:j...@8bytes.org] > > Sent: Wednesday, June 24, 2015 11:46 PM > > To: Alex Williamson > > Cc: Wu, Feng; Eric Auger; Avi Kivity; kvm@vger.kernel.org; > > linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com > > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > > On Thu, Jun 18, 2015 at 02:04:08PM -0600, Alex Williamson wrote: > > > There are plenty of details to be filled in, > > > > I also need to fill plenty of details in my head first, so here are some > > suggestions based on my current understanding. Please don't hesitate to > > correct me if where I got something wrong. > > > > So first I totally agree that the handling of PI/non-PI configurations > > should be transparent to user-space. > > After thinking about this a bit more, I recall that why I used user-space > to trigger the IRTE update for posted-interrupts, here is the reason: > > Let's take MSI for an example: > When guest updates the MSI configuration, here is the code path in > QEMU and KVM: > > vfio_update_msi() --> vfio_update_kvm_msi_virq() --> > kvm_irqchip_update_msi_route() --> kvm_update_routing_entry() --> > kvm_irqchip_commit_routes() --> kvm_irqchip_commit_routes() --> > KVM_SET_GSI_ROUTING --> kvm_set_irq_routing() > > It will finally go to kvm_set_irq_routing() in KVM, there are two problem: > 1. It use RCU in this function, it is hard to find which entry in the irq > routing > table is being updated. > 2. Even we find the updated entry, it is hard to find the associated assigned > device with this irq routing entry. > > So I used a VFIO API to notify KVM the updated MSI/MSIx configuration and > the associated assigned devices. I think we need to find a way to address > the above two issues before going forward. Alex, what is your opinion? So the trouble is that QEMU vfio updates a single MSI vector, but that just updates a single entry within a whole table of routes, then the whole table is pushed to KVM. But in kvm_set_irq_routing() we have access to both the new and the old tables, so we do have the ability to detect the change. We can therefore detect which GSI changed and cross-reference that to KVMs irqfds. If we have an irqfd that matches the GSI then we have all the information we need, right? We can use the eventfd_ctx of the irqfd to call into the IRQ bypass manager if we need to. If it's an irqfd that's already enabled for bypass then we may already have the data we need to tweak the PI config. Yes, I agree it's more difficult, but it doesn't appear to be impossible, right? Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Joerg Roedel [mailto:j...@8bytes.org]
> Sent: Wednesday, June 24, 2015 11:46 PM
> To: Alex Williamson
> Cc: Wu, Feng; Eric Auger; Avi Kivity; kvm@vger.kernel.org;
> linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> On Thu, Jun 18, 2015 at 02:04:08PM -0600, Alex Williamson wrote:
> > There are plenty of details to be filled in,
>
> I also need to fill plenty of details in my head first, so here are some
> suggestions based on my current understanding. Please don't hesitate to
> correct me if where I got something wrong.
>
> So first I totally agree that the handling of PI/non-PI configurations
> should be transparent to user-space.
After thinking about this a bit more, I recall that why I used user-space
to trigger the IRTE update for posted-interrupts, here is the reason:
Let's take MSI for an example:
When guest updates the MSI configuration, here is the code path in
QEMU and KVM:
vfio_update_msi() --> vfio_update_kvm_msi_virq() -->
kvm_irqchip_update_msi_route() --> kvm_update_routing_entry() -->
kvm_irqchip_commit_routes() --> kvm_irqchip_commit_routes() -->
KVM_SET_GSI_ROUTING --> kvm_set_irq_routing()
It will finally go to kvm_set_irq_routing() in KVM, there are two problem:
1. It use RCU in this function, it is hard to find which entry in the irq
routing
table is being updated.
2. Even we find the updated entry, it is hard to find the associated assigned
device with this irq routing entry.
So I used a VFIO API to notify KVM the updated MSI/MSIx configuration and
the associated assigned devices. I think we need to find a way to address
the above two issues before going forward. Alex, what is your opinion?
Thanks a lot!
Thanks,
Feng
>
> I read a bit through the VT-d spec, and my understanding of posted
> interrupts so far is that:
>
> 1) Each VCPU gets a PI-Descriptor with its pending Posted
> Interrupts. This descriptor needs to be updated when a VCPU
> is migrated to another PCPU and should thus be under control
> of KVM.
>
> This is similar to the vAPIC backing page in the AMD version
> of this, except that the PCPU routing information is stored
> somewhere else on AMD.
>
> 2) As long as the VCPU runs the IRTEs are configured for
> posting, when the VCPU goes to sleep the old remapped entry is
> established again. So when the VCPU sleeps the interrupt
> would get routed to VFIO and forwarded through the eventfd.
>
> This would be different to the AMD version, where we have a
> running bit. When this is clear the IOMMU will trigger an event
> in its event-log. This might need special handling in VFIO
> ('might' because VFIO does not need to forward the interrupt,
> it just needs to make sure the VCPU wakes up).
>
> Please correct me if my understanding of the Intel version is
> wrong.
>
> So most of the data structures the IOMMU reads for this need to be
> updated from KVM code (either x86-generic or AMD/Intel specific code),
> as KVM has the information about VCPU load/unload and the IRQ routing.
>
> What KVM needs from VFIO are the informations about the physical
> interrupts, and it makes total sense to attach them as metadata to the
> eventfd.
>
> But the problems start at how this metadata should look like. It would
> be good to have some generic description, but not sure if this is
> possible. Otherwise this metadata would need to be requested by VFIO
> from the IOMMU driver and passed on to KVM, which it then passes back to
> the IOMMU driver. Or something like that.
>
>
>
> Joerg
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Alex Williamson [mailto:alex.william...@redhat.com]
> Sent: Thursday, June 25, 2015 3:49 AM
> To: Eric Auger
> Cc: Joerg Roedel; Avi Kivity; Wu, Feng; kvm@vger.kernel.org;
> linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> On Wed, 2015-06-24 at 18:25 +0200, Eric Auger wrote:
> > Hi Joerg,
> >
> > On 06/24/2015 05:50 PM, Joerg Roedel wrote:
> > > On Mon, Jun 15, 2015 at 06:17:03PM +0200, Eric Auger wrote:
> > >> I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO
> > >> IRQ forward control" series? Or is that "central registry maintained by
> > >> a posted interrupts manager" something more specific to x86?
> > >
> > > From what I understood so far, the feature you implemented for ARM is a
> > > bit different from the ones that get introduced to x86.
> > >
> > > Can you please share some details on how the ARM version works? I am
> > > interested in how the GICv2 is configured for IRQ forwarding. The
> > > question is whether the forwarding information needs to be updated from
> > > KVM and what information about the IRQ KVM needs for this.
> >
> > The principle is that when you inject a virtual IRQ to a guest, you
> > program a register in the GIC, known as a list register. There you put
> > both the virtual IRQ you want to inject but also the physical IRQ it is
> > linked with (HWbit mode set = forwarding set). When the guest completes
> > the virtual IRQ the GIC HW automatically deactivates the physical IRQ
> > found in the list register. In that mode the physical IRQ deactivation
> > is under the ownership of the guest (actually automatically done by the HW).
> >
> > If HWbit mode is *not* set (forwarding not set), you do not specify the
> > HW IRQ in the list register. The host deactivates the physical IRQ &
> > masks it before triggering the virtual IRQ. Only the virtual IRQ ID is
> > programmed in the list register. When the guest completes the virtual
> > IRQ, a physical maintenance IRQ is triggered. The hyp mode is entered
> > and eventually the host unmasks the IRQ.
> >
> > Some illustrations can be found in
> > http://www.linux-kvm.org/images/a/a8/01x04-ARMdevice.pdf
>
>
> I think an important aspect for our design is that in the case of Posted
> Interrupts, they're only used for edge triggered interrupts so VFIO is
> only an information provider for KVM to configure it.
Exactly! For PI, KVM only needs some information from VFIO when the
guests set the irq affinity.
Thanks,
Feng
VFIO will
> hopefully just see fewer interrupts as they magically appear directly in
> the guest. IRQ Forwarding however affects the de-assertion of level
> triggered interrupts. VFIO needs to switch to something more like an
> edge handler when IRQ Forwarding is enabled. So in that model, VFIO
> needs to provide information as well as consume it to change behavior.
> Thanks,
>
> Alex
N�r��yb�X��ǧv�^�){.n�+h����ܨ}���Ơz�&j:+v���zZ+��+zf���h���~i���z� �w���?�&�)ߢf
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Joerg Roedel [mailto:j...@8bytes.org]
> Sent: Wednesday, June 24, 2015 11:46 PM
> To: Alex Williamson
> Cc: Wu, Feng; Eric Auger; Avi Kivity; kvm@vger.kernel.org;
> linux-ker...@vger.kernel.org; pbonz...@redhat.com; mtosa...@redhat.com
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> On Thu, Jun 18, 2015 at 02:04:08PM -0600, Alex Williamson wrote:
> > There are plenty of details to be filled in,
>
> I also need to fill plenty of details in my head first, so here are some
> suggestions based on my current understanding. Please don't hesitate to
> correct me if where I got something wrong.
>
> So first I totally agree that the handling of PI/non-PI configurations
> should be transparent to user-space.
>
> I read a bit through the VT-d spec, and my understanding of posted
> interrupts so far is that:
>
> 1) Each VCPU gets a PI-Descriptor with its pending Posted
> Interrupts. This descriptor needs to be updated when a VCPU
> is migrated to another PCPU and should thus be under control
> of KVM.
>
> This is similar to the vAPIC backing page in the AMD version
> of this, except that the PCPU routing information is stored
> somewhere else on AMD.
>
> 2) As long as the VCPU runs the IRTEs are configured for
> posting, when the VCPU goes to sleep the old remapped entry is
> established again. So when the VCPU sleeps the interrupt
> would get routed to VFIO and forwarded through the eventfd.
When the vCPU sleeps, says, blocked when guest is running HLT, the
interrupt is still in posted mode. The solution is when the vCPU is blocked,
we use another notification vector (named wakeup notification vector) to
wakeup the blocked vCPU when interrupts happens. And in the wakeup
event handler, we unblock the vCPU.
Thanks,
Feng
>
> This would be different to the AMD version, where we have a
> running bit. When this is clear the IOMMU will trigger an event
> in its event-log. This might need special handling in VFIO
> ('might' because VFIO does not need to forward the interrupt,
> it just needs to make sure the VCPU wakes up).
>
> Please correct me if my understanding of the Intel version is
> wrong.
>
> So most of the data structures the IOMMU reads for this need to be
> updated from KVM code (either x86-generic or AMD/Intel specific code),
> as KVM has the information about VCPU load/unload and the IRQ routing.
Yes, this part has nothing to do with VFIO, KVM itself can handle it well.
>
> What KVM needs from VFIO are the informations about the physical
> interrupts, and it makes total sense to attach them as metadata to the
> eventfd.
When guest set the irq affinity, QEMU first gets the MSI/MSIx configuration,
then it passes these information to kernel space via VFIO infrastructure, we
need these MSI/MSIx configuration to update the associated posted-format
IRTE according. This is the key point for PI in term of VFIO.
Thanks,
Feng
>
> But the problems start at how this metadata should look like. It would
> be good to have some generic description, but not sure if this is
> possible. Otherwise this metadata would need to be requested by VFIO
> from the IOMMU driver and passed on to KVM, which it then passes back to
> the IOMMU driver. Or something like that.
>
>
>
> Joerg
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Wed, 2015-06-24 at 18:25 +0200, Eric Auger wrote: > Hi Joerg, > > On 06/24/2015 05:50 PM, Joerg Roedel wrote: > > On Mon, Jun 15, 2015 at 06:17:03PM +0200, Eric Auger wrote: > >> I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO > >> IRQ forward control" series? Or is that "central registry maintained by > >> a posted interrupts manager" something more specific to x86? > > > > From what I understood so far, the feature you implemented for ARM is a > > bit different from the ones that get introduced to x86. > > > > Can you please share some details on how the ARM version works? I am > > interested in how the GICv2 is configured for IRQ forwarding. The > > question is whether the forwarding information needs to be updated from > > KVM and what information about the IRQ KVM needs for this. > > The principle is that when you inject a virtual IRQ to a guest, you > program a register in the GIC, known as a list register. There you put > both the virtual IRQ you want to inject but also the physical IRQ it is > linked with (HWbit mode set = forwarding set). When the guest completes > the virtual IRQ the GIC HW automatically deactivates the physical IRQ > found in the list register. In that mode the physical IRQ deactivation > is under the ownership of the guest (actually automatically done by the HW). > > If HWbit mode is *not* set (forwarding not set), you do not specify the > HW IRQ in the list register. The host deactivates the physical IRQ & > masks it before triggering the virtual IRQ. Only the virtual IRQ ID is > programmed in the list register. When the guest completes the virtual > IRQ, a physical maintenance IRQ is triggered. The hyp mode is entered > and eventually the host unmasks the IRQ. > > Some illustrations can be found in > http://www.linux-kvm.org/images/a/a8/01x04-ARMdevice.pdf I think an important aspect for our design is that in the case of Posted Interrupts, they're only used for edge triggered interrupts so VFIO is only an information provider for KVM to configure it. VFIO will hopefully just see fewer interrupts as they magically appear directly in the guest. IRQ Forwarding however affects the de-assertion of level triggered interrupts. VFIO needs to switch to something more like an edge handler when IRQ Forwarding is enabled. So in that model, VFIO needs to provide information as well as consume it to change behavior. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
Hi Joerg, On 06/24/2015 05:50 PM, Joerg Roedel wrote: > On Mon, Jun 15, 2015 at 06:17:03PM +0200, Eric Auger wrote: >> I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO >> IRQ forward control" series? Or is that "central registry maintained by >> a posted interrupts manager" something more specific to x86? > > From what I understood so far, the feature you implemented for ARM is a > bit different from the ones that get introduced to x86. > > Can you please share some details on how the ARM version works? I am > interested in how the GICv2 is configured for IRQ forwarding. The > question is whether the forwarding information needs to be updated from > KVM and what information about the IRQ KVM needs for this. The principle is that when you inject a virtual IRQ to a guest, you program a register in the GIC, known as a list register. There you put both the virtual IRQ you want to inject but also the physical IRQ it is linked with (HWbit mode set = forwarding set). When the guest completes the virtual IRQ the GIC HW automatically deactivates the physical IRQ found in the list register. In that mode the physical IRQ deactivation is under the ownership of the guest (actually automatically done by the HW). If HWbit mode is *not* set (forwarding not set), you do not specify the HW IRQ in the list register. The host deactivates the physical IRQ & masks it before triggering the virtual IRQ. Only the virtual IRQ ID is programmed in the list register. When the guest completes the virtual IRQ, a physical maintenance IRQ is triggered. The hyp mode is entered and eventually the host unmasks the IRQ. Some illustrations can be found in http://www.linux-kvm.org/images/a/a8/01x04-ARMdevice.pdf Hope it helps Eric > > > Joerg > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Mon, Jun 15, 2015 at 06:17:03PM +0200, Eric Auger wrote: > I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO > IRQ forward control" series? Or is that "central registry maintained by > a posted interrupts manager" something more specific to x86? >From what I understood so far, the feature you implemented for ARM is a bit different from the ones that get introduced to x86. Can you please share some details on how the ARM version works? I am interested in how the GICv2 is configured for IRQ forwarding. The question is whether the forwarding information needs to be updated from KVM and what information about the IRQ KVM needs for this. Joerg -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Thu, Jun 18, 2015 at 02:04:08PM -0600, Alex Williamson wrote:
> There are plenty of details to be filled in,
I also need to fill plenty of details in my head first, so here are some
suggestions based on my current understanding. Please don't hesitate to
correct me if where I got something wrong.
So first I totally agree that the handling of PI/non-PI configurations
should be transparent to user-space.
I read a bit through the VT-d spec, and my understanding of posted
interrupts so far is that:
1) Each VCPU gets a PI-Descriptor with its pending Posted
Interrupts. This descriptor needs to be updated when a VCPU
is migrated to another PCPU and should thus be under control
of KVM.
This is similar to the vAPIC backing page in the AMD version
of this, except that the PCPU routing information is stored
somewhere else on AMD.
2) As long as the VCPU runs the IRTEs are configured for
posting, when the VCPU goes to sleep the old remapped entry is
established again. So when the VCPU sleeps the interrupt
would get routed to VFIO and forwarded through the eventfd.
This would be different to the AMD version, where we have a
running bit. When this is clear the IOMMU will trigger an event
in its event-log. This might need special handling in VFIO
('might' because VFIO does not need to forward the interrupt,
it just needs to make sure the VCPU wakes up).
Please correct me if my understanding of the Intel version is
wrong.
So most of the data structures the IOMMU reads for this need to be
updated from KVM code (either x86-generic or AMD/Intel specific code),
as KVM has the information about VCPU load/unload and the IRQ routing.
What KVM needs from VFIO are the informations about the physical
interrupts, and it makes total sense to attach them as metadata to the
eventfd.
But the problems start at how this metadata should look like. It would
be good to have some generic description, but not sure if this is
possible. Otherwise this metadata would need to be requested by VFIO
from the IOMMU driver and passed on to KVM, which it then passes back to
the IOMMU driver. Or something like that.
Joerg
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
[Adding Joerg since he was part of this original idea] On Thu, 2015-06-18 at 09:16 +, Wu, Feng wrote: > > > > -Original Message- > > From: Alex Williamson [mailto:alex.william...@redhat.com] > > Sent: Tuesday, June 16, 2015 12:45 AM > > To: Eric Auger > > Cc: Avi Kivity; Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org; > > pbonz...@redhat.com; mtosa...@redhat.com > > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > > On Mon, 2015-06-15 at 18:17 +0200, Eric Auger wrote: > > > Hi Alex, all, > > > On 06/12/2015 09:03 PM, Alex Williamson wrote: > > > > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: > > > >> On 06/12/2015 06:41 PM, Alex Williamson wrote: > > > >>> On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > > > >>>>> -Original Message- > > > >>>>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] > > > >>>>> Sent: Friday, June 12, 2015 3:59 AM > > > >>>>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > > > >>>>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; > > > >>>>> alex.william...@redhat.com; eric.au...@linaro.org > > > >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > >>>>> > > > >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote: > > > >>>>>> From: Eric Auger > > > >>>>>> > > > >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE > > group > > > >>>>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > > > >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be > > able > > > >>>>>> to set a VFIO device IRQ as forwarded or not forwarded. > > > >>>>>> the command takes as argument a handle to a new struct named > > > >>>>>> kvm_vfio_dev_irq. > > > >>>>> Is there no way to do this automatically? After all, vfio knows > > > >>>>> that a > > > >>>>> device interrupt is forwarded to some eventfd, and kvm knows that > > some > > > >>>>> eventfd is forwarded to a guest interrupt. If they compare notes > > > >>>>> through a central registry, they can figure out that the interrupt > > > >>>>> needs > > > >>>>> to be forwarded. > > > >>>> Oh, just like Eric mentioned in his reply, this description is out > > > >>>> of context > > of > > > >>>> this series, I will remove them in the next version. > > > >>> > > > >>> I suspect Avi's question was more general. While forward/unforward is > > > >>> out of context for this series, it's very similar in nature to > > > >>> enabling/disabling posted interrupts. So I think the question remains > > > >>> whether we really need userspace to participate in creating this > > > >>> shortcut or if kvm and vfio can some how orchestrate figuring it out > > > >>> automatically. > > > >>> > > > >>> Personally I don't know how we could do it automatically. We've > > > >>> always > > > >>> relied on userspace to independently setup vfio and kvm such that > > > >>> neither have any idea that the other is there and update each side > > > >>> independently when anything changes. So it seems consistent to > > continue > > > >>> that here. It doesn't seem like there's much to gain performance-wise > > > >>> either, updates should be a relatively rare event I'd expect. > > > >>> > > > >>> There's really no metadata associated with an eventfd, so "comparing > > > >>> notes" automatically might imply some central registration entity. > > > >>> That > > > >>> immediately sounds like a much more complex solution, but maybe Avi > > has > > > >>> some ideas to manage it. Thanks, > > > >>> > > > >> > > > >> The idea is to have a central registry maintained by a posted > > > >> interrupts > > > >> manager. Both vfio and kvm pass the filp (a
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message- > From: Alex Williamson [mailto:alex.william...@redhat.com] > Sent: Tuesday, June 16, 2015 12:45 AM > To: Eric Auger > Cc: Avi Kivity; Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org; > pbonz...@redhat.com; mtosa...@redhat.com > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > On Mon, 2015-06-15 at 18:17 +0200, Eric Auger wrote: > > Hi Alex, all, > > On 06/12/2015 09:03 PM, Alex Williamson wrote: > > > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: > > >> On 06/12/2015 06:41 PM, Alex Williamson wrote: > > >>> On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > > >>>>> -Original Message- > > >>>>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] > > >>>>> Sent: Friday, June 12, 2015 3:59 AM > > >>>>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > > >>>>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; > > >>>>> alex.william...@redhat.com; eric.au...@linaro.org > > >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > >>>>> > > >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote: > > >>>>>> From: Eric Auger > > >>>>>> > > >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE > group > > >>>>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > > >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be > able > > >>>>>> to set a VFIO device IRQ as forwarded or not forwarded. > > >>>>>> the command takes as argument a handle to a new struct named > > >>>>>> kvm_vfio_dev_irq. > > >>>>> Is there no way to do this automatically? After all, vfio knows that > > >>>>> a > > >>>>> device interrupt is forwarded to some eventfd, and kvm knows that > some > > >>>>> eventfd is forwarded to a guest interrupt. If they compare notes > > >>>>> through a central registry, they can figure out that the interrupt > > >>>>> needs > > >>>>> to be forwarded. > > >>>> Oh, just like Eric mentioned in his reply, this description is out of > > >>>> context > of > > >>>> this series, I will remove them in the next version. > > >>> > > >>> I suspect Avi's question was more general. While forward/unforward is > > >>> out of context for this series, it's very similar in nature to > > >>> enabling/disabling posted interrupts. So I think the question remains > > >>> whether we really need userspace to participate in creating this > > >>> shortcut or if kvm and vfio can some how orchestrate figuring it out > > >>> automatically. > > >>> > > >>> Personally I don't know how we could do it automatically. We've always > > >>> relied on userspace to independently setup vfio and kvm such that > > >>> neither have any idea that the other is there and update each side > > >>> independently when anything changes. So it seems consistent to > continue > > >>> that here. It doesn't seem like there's much to gain performance-wise > > >>> either, updates should be a relatively rare event I'd expect. > > >>> > > >>> There's really no metadata associated with an eventfd, so "comparing > > >>> notes" automatically might imply some central registration entity. That > > >>> immediately sounds like a much more complex solution, but maybe Avi > has > > >>> some ideas to manage it. Thanks, > > >>> > > >> > > >> The idea is to have a central registry maintained by a posted interrupts > > >> manager. Both vfio and kvm pass the filp (along with extra information) > > >> to the posted interrupts manager, which, when it detects a filp match, > > >> tells each of them what to do. > > >> > > >> The advantages are: > > >> - old userspace gains the optimization without change > > >> - a userspace API is more expensive to maintain than internal kernel > > >> interfaces (CVEs, documentation, maintaining backwards compatibility) > > >> - if you can do it without a new interface, this indicat
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Mon, 2015-06-15 at 18:17 +0200, Eric Auger wrote: > Hi Alex, all, > On 06/12/2015 09:03 PM, Alex Williamson wrote: > > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: > >> On 06/12/2015 06:41 PM, Alex Williamson wrote: > >>> On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > >>>>> -Original Message- > >>>>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] > >>>>> Sent: Friday, June 12, 2015 3:59 AM > >>>>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > >>>>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; > >>>>> alex.william...@redhat.com; eric.au...@linaro.org > >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > >>>>> > >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote: > >>>>>> From: Eric Auger > >>>>>> > >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE group > >>>>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able > >>>>>> to set a VFIO device IRQ as forwarded or not forwarded. > >>>>>> the command takes as argument a handle to a new struct named > >>>>>> kvm_vfio_dev_irq. > >>>>> Is there no way to do this automatically? After all, vfio knows that a > >>>>> device interrupt is forwarded to some eventfd, and kvm knows that some > >>>>> eventfd is forwarded to a guest interrupt. If they compare notes > >>>>> through a central registry, they can figure out that the interrupt needs > >>>>> to be forwarded. > >>>> Oh, just like Eric mentioned in his reply, this description is out of > >>>> context of > >>>> this series, I will remove them in the next version. > >>> > >>> I suspect Avi's question was more general. While forward/unforward is > >>> out of context for this series, it's very similar in nature to > >>> enabling/disabling posted interrupts. So I think the question remains > >>> whether we really need userspace to participate in creating this > >>> shortcut or if kvm and vfio can some how orchestrate figuring it out > >>> automatically. > >>> > >>> Personally I don't know how we could do it automatically. We've always > >>> relied on userspace to independently setup vfio and kvm such that > >>> neither have any idea that the other is there and update each side > >>> independently when anything changes. So it seems consistent to continue > >>> that here. It doesn't seem like there's much to gain performance-wise > >>> either, updates should be a relatively rare event I'd expect. > >>> > >>> There's really no metadata associated with an eventfd, so "comparing > >>> notes" automatically might imply some central registration entity. That > >>> immediately sounds like a much more complex solution, but maybe Avi has > >>> some ideas to manage it. Thanks, > >>> > >> > >> The idea is to have a central registry maintained by a posted interrupts > >> manager. Both vfio and kvm pass the filp (along with extra information) > >> to the posted interrupts manager, which, when it detects a filp match, > >> tells each of them what to do. > >> > >> The advantages are: > >> - old userspace gains the optimization without change > >> - a userspace API is more expensive to maintain than internal kernel > >> interfaces (CVEs, documentation, maintaining backwards compatibility) > >> - if you can do it without a new interface, this indicates that all the > >> information in the new interface is redundant. That means you have to > >> check it for consistency with the existing information, so it's extra > >> work (likely, it's exactly what the posted interrupt manager would be > >> doing anyway). > > > > Yep, those all sound like good things and I believe that's similar in > > design to the way we had originally discussed this interaction at > > LPC/KVM Forum several years ago. I'd be in favor of that approach. > > I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO > IRQ forward control" series? Or is that "central registry maintained by > a posted interrupts m
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
Hi Alex, all, On 06/12/2015 09:03 PM, Alex Williamson wrote: > On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: >> On 06/12/2015 06:41 PM, Alex Williamson wrote: >>> On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: >>>>> -Original Message- >>>>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] >>>>> Sent: Friday, June 12, 2015 3:59 AM >>>>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org >>>>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; >>>>> alex.william...@redhat.com; eric.au...@linaro.org >>>>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding >>>>> >>>>> On 06/11/2015 01:51 PM, Feng Wu wrote: >>>>>> From: Eric Auger >>>>>> >>>>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE group >>>>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, >>>>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able >>>>>> to set a VFIO device IRQ as forwarded or not forwarded. >>>>>> the command takes as argument a handle to a new struct named >>>>>> kvm_vfio_dev_irq. >>>>> Is there no way to do this automatically? After all, vfio knows that a >>>>> device interrupt is forwarded to some eventfd, and kvm knows that some >>>>> eventfd is forwarded to a guest interrupt. If they compare notes >>>>> through a central registry, they can figure out that the interrupt needs >>>>> to be forwarded. >>>> Oh, just like Eric mentioned in his reply, this description is out of >>>> context of >>>> this series, I will remove them in the next version. >>> >>> I suspect Avi's question was more general. While forward/unforward is >>> out of context for this series, it's very similar in nature to >>> enabling/disabling posted interrupts. So I think the question remains >>> whether we really need userspace to participate in creating this >>> shortcut or if kvm and vfio can some how orchestrate figuring it out >>> automatically. >>> >>> Personally I don't know how we could do it automatically. We've always >>> relied on userspace to independently setup vfio and kvm such that >>> neither have any idea that the other is there and update each side >>> independently when anything changes. So it seems consistent to continue >>> that here. It doesn't seem like there's much to gain performance-wise >>> either, updates should be a relatively rare event I'd expect. >>> >>> There's really no metadata associated with an eventfd, so "comparing >>> notes" automatically might imply some central registration entity. That >>> immediately sounds like a much more complex solution, but maybe Avi has >>> some ideas to manage it. Thanks, >>> >> >> The idea is to have a central registry maintained by a posted interrupts >> manager. Both vfio and kvm pass the filp (along with extra information) >> to the posted interrupts manager, which, when it detects a filp match, >> tells each of them what to do. >> >> The advantages are: >> - old userspace gains the optimization without change >> - a userspace API is more expensive to maintain than internal kernel >> interfaces (CVEs, documentation, maintaining backwards compatibility) >> - if you can do it without a new interface, this indicates that all the >> information in the new interface is redundant. That means you have to >> check it for consistency with the existing information, so it's extra >> work (likely, it's exactly what the posted interrupt manager would be >> doing anyway). > > Yep, those all sound like good things and I believe that's similar in > design to the way we had originally discussed this interaction at > LPC/KVM Forum several years ago. I'd be in favor of that approach. I guess this discussion also is relevant wrt "[RFC v6 00/16] KVM-VFIO IRQ forward control" series? Or is that "central registry maintained by a posted interrupts manager" something more specific to x86? Thank you in advance Best Regards Eric > Thanks, > > Alex > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Alex Williamson [mailto:alex.william...@redhat.com]
> Sent: Saturday, June 13, 2015 3:04 AM
> To: Avi Kivity
> Cc: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org;
> pbonz...@redhat.com; mtosa...@redhat.com; eric.au...@linaro.org
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote:
> > On 06/12/2015 06:41 PM, Alex Williamson wrote:
> > > On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote:
> > >>> -Original Message-
> > >>> From: Avi Kivity [mailto:avi.kiv...@gmail.com]
> > >>> Sent: Friday, June 12, 2015 3:59 AM
> > >>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org
> > >>> Cc: pbonz...@redhat.com; mtosa...@redhat.com;
> > >>> alex.william...@redhat.com; eric.au...@linaro.org
> > >>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> > >>>
> > >>> On 06/11/2015 01:51 PM, Feng Wu wrote:
> > >>>> From: Eric Auger
> > >>>>
> > >>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
> > >>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
> > >>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
> > >>>> to set a VFIO device IRQ as forwarded or not forwarded.
> > >>>> the command takes as argument a handle to a new struct named
> > >>>> kvm_vfio_dev_irq.
> > >>> Is there no way to do this automatically? After all, vfio knows that a
> > >>> device interrupt is forwarded to some eventfd, and kvm knows that some
> > >>> eventfd is forwarded to a guest interrupt. If they compare notes
> > >>> through a central registry, they can figure out that the interrupt needs
> > >>> to be forwarded.
> > >> Oh, just like Eric mentioned in his reply, this description is out of
> > >> context of
> > >> this series, I will remove them in the next version.
> > >
> > > I suspect Avi's question was more general. While forward/unforward is
> > > out of context for this series, it's very similar in nature to
> > > enabling/disabling posted interrupts. So I think the question remains
> > > whether we really need userspace to participate in creating this
> > > shortcut or if kvm and vfio can some how orchestrate figuring it out
> > > automatically.
> > >
> > > Personally I don't know how we could do it automatically. We've always
> > > relied on userspace to independently setup vfio and kvm such that
> > > neither have any idea that the other is there and update each side
> > > independently when anything changes. So it seems consistent to continue
> > > that here. It doesn't seem like there's much to gain performance-wise
> > > either, updates should be a relatively rare event I'd expect.
> > >
> > > There's really no metadata associated with an eventfd, so "comparing
> > > notes" automatically might imply some central registration entity. That
> > > immediately sounds like a much more complex solution, but maybe Avi has
> > > some ideas to manage it. Thanks,
> > >
> >
> > The idea is to have a central registry maintained by a posted interrupts
> > manager. Both vfio and kvm pass the filp (along with extra information)
> > to the posted interrupts manager, which, when it detects a filp match,
> > tells each of them what to do.
> >
> > The advantages are:
> > - old userspace gains the optimization without change
> > - a userspace API is more expensive to maintain than internal kernel
> > interfaces (CVEs, documentation, maintaining backwards compatibility)
> > - if you can do it without a new interface, this indicates that all the
> > information in the new interface is redundant. That means you have to
> > check it for consistency with the existing information, so it's extra
> > work (likely, it's exactly what the posted interrupt manager would be
> > doing anyway).
>
> Yep, those all sound like good things and I believe that's similar in
> design to the way we had originally discussed this interaction at
> LPC/KVM Forum several years ago. I'd be in favor of that approach.
> Thanks,
This seems a little complex compared to the current solution, since I am
not quite familiar with VFIO, Alex, can you help on this if we need to do this
that way, especially for the VFIO part?
Thanks,
Feng
>
> Alex
N�r��yb�X��ǧv�^�){.n�+h����ܨ}���Ơz�&j:+v���zZ+��+zf���h���~i���z� �w���?�&�)ߢf
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Fri, 2015-06-12 at 21:48 +0300, Avi Kivity wrote: > On 06/12/2015 06:41 PM, Alex Williamson wrote: > > On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > >>> -Original Message- > >>> From: Avi Kivity [mailto:avi.kiv...@gmail.com] > >>> Sent: Friday, June 12, 2015 3:59 AM > >>> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > >>> Cc: pbonz...@redhat.com; mtosa...@redhat.com; > >>> alex.william...@redhat.com; eric.au...@linaro.org > >>> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > >>> > >>> On 06/11/2015 01:51 PM, Feng Wu wrote: > >>>> From: Eric Auger > >>>> > >>>> This patch adds and documents a new KVM_DEV_VFIO_DEVICE group > >>>> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > >>>> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able > >>>> to set a VFIO device IRQ as forwarded or not forwarded. > >>>> the command takes as argument a handle to a new struct named > >>>> kvm_vfio_dev_irq. > >>> Is there no way to do this automatically? After all, vfio knows that a > >>> device interrupt is forwarded to some eventfd, and kvm knows that some > >>> eventfd is forwarded to a guest interrupt. If they compare notes > >>> through a central registry, they can figure out that the interrupt needs > >>> to be forwarded. > >> Oh, just like Eric mentioned in his reply, this description is out of > >> context of > >> this series, I will remove them in the next version. > > > > I suspect Avi's question was more general. While forward/unforward is > > out of context for this series, it's very similar in nature to > > enabling/disabling posted interrupts. So I think the question remains > > whether we really need userspace to participate in creating this > > shortcut or if kvm and vfio can some how orchestrate figuring it out > > automatically. > > > > Personally I don't know how we could do it automatically. We've always > > relied on userspace to independently setup vfio and kvm such that > > neither have any idea that the other is there and update each side > > independently when anything changes. So it seems consistent to continue > > that here. It doesn't seem like there's much to gain performance-wise > > either, updates should be a relatively rare event I'd expect. > > > > There's really no metadata associated with an eventfd, so "comparing > > notes" automatically might imply some central registration entity. That > > immediately sounds like a much more complex solution, but maybe Avi has > > some ideas to manage it. Thanks, > > > > The idea is to have a central registry maintained by a posted interrupts > manager. Both vfio and kvm pass the filp (along with extra information) > to the posted interrupts manager, which, when it detects a filp match, > tells each of them what to do. > > The advantages are: > - old userspace gains the optimization without change > - a userspace API is more expensive to maintain than internal kernel > interfaces (CVEs, documentation, maintaining backwards compatibility) > - if you can do it without a new interface, this indicates that all the > information in the new interface is redundant. That means you have to > check it for consistency with the existing information, so it's extra > work (likely, it's exactly what the posted interrupt manager would be > doing anyway). Yep, those all sound like good things and I believe that's similar in design to the way we had originally discussed this interaction at LPC/KVM Forum several years ago. I'd be in favor of that approach. Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On 06/12/2015 06:41 PM, Alex Williamson wrote: On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: -Original Message- From: Avi Kivity [mailto:avi.kiv...@gmail.com] Sent: Friday, June 12, 2015 3:59 AM To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org Cc: pbonz...@redhat.com; mtosa...@redhat.com; alex.william...@redhat.com; eric.au...@linaro.org Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding On 06/11/2015 01:51 PM, Feng Wu wrote: From: Eric Auger This patch adds and documents a new KVM_DEV_VFIO_DEVICE group and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able to set a VFIO device IRQ as forwarded or not forwarded. the command takes as argument a handle to a new struct named kvm_vfio_dev_irq. Is there no way to do this automatically? After all, vfio knows that a device interrupt is forwarded to some eventfd, and kvm knows that some eventfd is forwarded to a guest interrupt. If they compare notes through a central registry, they can figure out that the interrupt needs to be forwarded. Oh, just like Eric mentioned in his reply, this description is out of context of this series, I will remove them in the next version. I suspect Avi's question was more general. While forward/unforward is out of context for this series, it's very similar in nature to enabling/disabling posted interrupts. So I think the question remains whether we really need userspace to participate in creating this shortcut or if kvm and vfio can some how orchestrate figuring it out automatically. Personally I don't know how we could do it automatically. We've always relied on userspace to independently setup vfio and kvm such that neither have any idea that the other is there and update each side independently when anything changes. So it seems consistent to continue that here. It doesn't seem like there's much to gain performance-wise either, updates should be a relatively rare event I'd expect. There's really no metadata associated with an eventfd, so "comparing notes" automatically might imply some central registration entity. That immediately sounds like a much more complex solution, but maybe Avi has some ideas to manage it. Thanks, The idea is to have a central registry maintained by a posted interrupts manager. Both vfio and kvm pass the filp (along with extra information) to the posted interrupts manager, which, when it detects a filp match, tells each of them what to do. The advantages are: - old userspace gains the optimization without change - a userspace API is more expensive to maintain than internal kernel interfaces (CVEs, documentation, maintaining backwards compatibility) - if you can do it without a new interface, this indicates that all the information in the new interface is redundant. That means you have to check it for consistency with the existing information, so it's extra work (likely, it's exactly what the posted interrupt manager would be doing anyway). -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On Fri, 2015-06-12 at 00:23 +, Wu, Feng wrote: > > > -Original Message- > > From: Avi Kivity [mailto:avi.kiv...@gmail.com] > > Sent: Friday, June 12, 2015 3:59 AM > > To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org > > Cc: pbonz...@redhat.com; mtosa...@redhat.com; > > alex.william...@redhat.com; eric.au...@linaro.org > > Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding > > > > On 06/11/2015 01:51 PM, Feng Wu wrote: > > > From: Eric Auger > > > > > > This patch adds and documents a new KVM_DEV_VFIO_DEVICE group > > > and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ, > > > KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able > > > to set a VFIO device IRQ as forwarded or not forwarded. > > > the command takes as argument a handle to a new struct named > > > kvm_vfio_dev_irq. > > > > Is there no way to do this automatically? After all, vfio knows that a > > device interrupt is forwarded to some eventfd, and kvm knows that some > > eventfd is forwarded to a guest interrupt. If they compare notes > > through a central registry, they can figure out that the interrupt needs > > to be forwarded. > > Oh, just like Eric mentioned in his reply, this description is out of context > of > this series, I will remove them in the next version. I suspect Avi's question was more general. While forward/unforward is out of context for this series, it's very similar in nature to enabling/disabling posted interrupts. So I think the question remains whether we really need userspace to participate in creating this shortcut or if kvm and vfio can some how orchestrate figuring it out automatically. Personally I don't know how we could do it automatically. We've always relied on userspace to independently setup vfio and kvm such that neither have any idea that the other is there and update each side independently when anything changes. So it seems consistent to continue that here. It doesn't seem like there's much to gain performance-wise either, updates should be a relatively rare event I'd expect. There's really no metadata associated with an eventfd, so "comparing notes" automatically might imply some central registration entity. That immediately sounds like a much more complex solution, but maybe Avi has some ideas to manage it. Thanks, Alex > > > Signed-off-by: Eric Auger > > > --- > > > Documentation/virtual/kvm/devices/vfio.txt | 34 > > -- > > > include/uapi/linux/kvm.h | 12 +++ > > > 2 files changed, 40 insertions(+), 6 deletions(-) > > > > > > diff --git a/Documentation/virtual/kvm/devices/vfio.txt > > b/Documentation/virtual/kvm/devices/vfio.txt > > > index ef51740..6186e6d 100644 > > > --- a/Documentation/virtual/kvm/devices/vfio.txt > > > +++ b/Documentation/virtual/kvm/devices/vfio.txt > > > @@ -4,15 +4,20 @@ VFIO virtual device > > > Device types supported: > > > KVM_DEV_TYPE_VFIO > > > > > > -Only one VFIO instance may be created per VM. The created device > > > -tracks VFIO groups in use by the VM and features of those groups > > > -important to the correctness and acceleration of the VM. As groups > > > -are enabled and disabled for use by the VM, KVM should be updated > > > -about their presence. When registered with KVM, a reference to the > > > -VFIO-group is held by KVM. > > > +Only one VFIO instance may be created per VM. > > > + > > > +The created device tracks VFIO groups in use by the VM and features > > > +of those groups important to the correctness and acceleration of > > > +the VM. As groups are enabled and disabled for use by the VM, KVM > > > +should be updated about their presence. When registered with KVM, > > > +a reference to the VFIO-group is held by KVM. > > > + > > > +The device also enables to control some IRQ settings of VFIO devices: > > > +forwarding/posting. > > > > > > Groups: > > > KVM_DEV_VFIO_GROUP > > > + KVM_DEV_VFIO_DEVICE > > > > > > KVM_DEV_VFIO_GROUP attributes: > > > KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device > > tracking > > > @@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes: > > > > > > For each, kvm_device_attr.addr points to an int32_t file descriptor > > > for the VFIO group. > > > + > > > +KVM_DEV_VFIO_DEVICE attributes: > > > + KVM_DEV_VFIO_DEVICE_FORWA
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Avi Kivity [mailto:avi.kiv...@gmail.com]
> Sent: Friday, June 12, 2015 3:59 AM
> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org
> Cc: pbonz...@redhat.com; mtosa...@redhat.com;
> alex.william...@redhat.com; eric.au...@linaro.org
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> On 06/11/2015 01:51 PM, Feng Wu wrote:
> > From: Eric Auger
> >
> > This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
> > and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
> > KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
> > to set a VFIO device IRQ as forwarded or not forwarded.
> > the command takes as argument a handle to a new struct named
> > kvm_vfio_dev_irq.
>
> Is there no way to do this automatically? After all, vfio knows that a
> device interrupt is forwarded to some eventfd, and kvm knows that some
> eventfd is forwarded to a guest interrupt. If they compare notes
> through a central registry, they can figure out that the interrupt needs
> to be forwarded.
Oh, just like Eric mentioned in his reply, this description is out of context of
this series, I will remove them in the next version.
Thanks,
Feng
>
>
> > Signed-off-by: Eric Auger
> > ---
> > Documentation/virtual/kvm/devices/vfio.txt | 34
> --
> > include/uapi/linux/kvm.h | 12 +++
> > 2 files changed, 40 insertions(+), 6 deletions(-)
> >
> > diff --git a/Documentation/virtual/kvm/devices/vfio.txt
> b/Documentation/virtual/kvm/devices/vfio.txt
> > index ef51740..6186e6d 100644
> > --- a/Documentation/virtual/kvm/devices/vfio.txt
> > +++ b/Documentation/virtual/kvm/devices/vfio.txt
> > @@ -4,15 +4,20 @@ VFIO virtual device
> > Device types supported:
> > KVM_DEV_TYPE_VFIO
> >
> > -Only one VFIO instance may be created per VM. The created device
> > -tracks VFIO groups in use by the VM and features of those groups
> > -important to the correctness and acceleration of the VM. As groups
> > -are enabled and disabled for use by the VM, KVM should be updated
> > -about their presence. When registered with KVM, a reference to the
> > -VFIO-group is held by KVM.
> > +Only one VFIO instance may be created per VM.
> > +
> > +The created device tracks VFIO groups in use by the VM and features
> > +of those groups important to the correctness and acceleration of
> > +the VM. As groups are enabled and disabled for use by the VM, KVM
> > +should be updated about their presence. When registered with KVM,
> > +a reference to the VFIO-group is held by KVM.
> > +
> > +The device also enables to control some IRQ settings of VFIO devices:
> > +forwarding/posting.
> >
> > Groups:
> > KVM_DEV_VFIO_GROUP
> > + KVM_DEV_VFIO_DEVICE
> >
> > KVM_DEV_VFIO_GROUP attributes:
> > KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device
> tracking
> > @@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes:
> >
> > For each, kvm_device_attr.addr points to an int32_t file descriptor
> > for the VFIO group.
> > +
> > +KVM_DEV_VFIO_DEVICE attributes:
> > + KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as
> forwarded
> > + KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as
> not forwarded
> > +
> > +For each, kvm_device_attr.addr points to a kvm_vfio_dev_irq struct.
> > +
> > +When forwarded, a physical IRQ is completed by the guest and not by the
> > +host. This requires HW support in the interrupt controller.
> > +
> > +Forwarding can only be set when the corresponding VFIO IRQ is not masked
> > +(would it be through VFIO_DEVICE_SET_IRQS command or as a consequence
> of this
> > +IRQ being currently handled) or active at interrupt controller level.
> > +In such a situation, -EAGAIN is returned. It is advised to to set the
> > +forwarding before the VFIO signaling is set up, this avoids trial and
> > errors.
> > +
> > +Unforwarding can happen at any time.
> > diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> > index 4b60056..798f3e4 100644
> > --- a/include/uapi/linux/kvm.h
> > +++ b/include/uapi/linux/kvm.h
> > @@ -999,6 +999,9 @@ struct kvm_device_attr {
> > #define KVM_DEV_VFIO_GROUP 1
> > #define KVM_DEV_VFIO_GROUP_ADD 1
> > #define KVM_DEV_VFIO_GROUP_DEL 2
> > +#define KVM_DEV_VFIO_DEVICE 2
> > +#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ
RE: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
> -Original Message-
> From: Eric Auger [mailto:eric.au...@linaro.org]
> Sent: Thursday, June 11, 2015 9:38 PM
> To: Wu, Feng; kvm@vger.kernel.org; linux-ker...@vger.kernel.org
> Cc: pbonz...@redhat.com; mtosa...@redhat.com;
> alex.william...@redhat.com
> Subject: Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
>
> Hi Feng,
> On 06/11/2015 12:51 PM, Feng Wu wrote:
> > From: Eric Auger
> >
> > This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
> > and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
> > KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
> > to set a VFIO device IRQ as forwarded or not forwarded.
> > the command takes as argument a handle to a new struct named
> > kvm_vfio_dev_irq.
> >
> > Signed-off-by: Eric Auger
> > ---
> > Documentation/virtual/kvm/devices/vfio.txt | 34
> --
> > include/uapi/linux/kvm.h | 12 +++
> > 2 files changed, 40 insertions(+), 6 deletions(-)
> >
> > diff --git a/Documentation/virtual/kvm/devices/vfio.txt
> b/Documentation/virtual/kvm/devices/vfio.txt
> > index ef51740..6186e6d 100644
> > --- a/Documentation/virtual/kvm/devices/vfio.txt
> > +++ b/Documentation/virtual/kvm/devices/vfio.txt
> > @@ -4,15 +4,20 @@ VFIO virtual device
> > Device types supported:
> >KVM_DEV_TYPE_VFIO
> >
> > -Only one VFIO instance may be created per VM. The created device
> > -tracks VFIO groups in use by the VM and features of those groups
> > -important to the correctness and acceleration of the VM. As groups
> > -are enabled and disabled for use by the VM, KVM should be updated
> > -about their presence. When registered with KVM, a reference to the
> > -VFIO-group is held by KVM.
> > +Only one VFIO instance may be created per VM.
> > +
> > +The created device tracks VFIO groups in use by the VM and features
> > +of those groups important to the correctness and acceleration of
> > +the VM. As groups are enabled and disabled for use by the VM, KVM
> > +should be updated about their presence. When registered with KVM,
> > +a reference to the VFIO-group is held by KVM.
> > +
> > +The device also enables to control some IRQ settings of VFIO devices:
> > +forwarding/posting.
> >
> > Groups:
> >KVM_DEV_VFIO_GROUP
> > + KVM_DEV_VFIO_DEVICE
> >
> > KVM_DEV_VFIO_GROUP attributes:
> >KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device
> tracking
> > @@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes:
> >
> > For each, kvm_device_attr.addr points to an int32_t file descriptor
> > for the VFIO group.
> > +
> > +KVM_DEV_VFIO_DEVICE attributes:
> > + KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as
> forwarded
> > + KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as
> not forwarded
> > +
> > +For each, kvm_device_attr.addr points to a kvm_vfio_dev_irq struct.
> > +
> > +When forwarded, a physical IRQ is completed by the guest and not by the
> > +host. This requires HW support in the interrupt controller.
> > +
> > +Forwarding can only be set when the corresponding VFIO IRQ is not masked
> > +(would it be through VFIO_DEVICE_SET_IRQS command or as a consequence
> of this
> > +IRQ being currently handled) or active at interrupt controller level.
> > +In such a situation, -EAGAIN is returned. It is advised to to set the
> > +forwarding before the VFIO signaling is set up, this avoids trial and
> > errors.
> > +
> > +Unforwarding can happen at any time.
> Unfortunately the above text is out of context of your series. If it
> simplifies things take the ownership of that patch file and remove my
> description of KVM_DEV_VFIO_DEVICE_FORWARD_IRQ.
Oh, I didn't notice that. It is a good suggestion, Eric!
Thanks,
Feng
> > diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> > index 4b60056..798f3e4 100644
> > --- a/include/uapi/linux/kvm.h
> > +++ b/include/uapi/linux/kvm.h
> > @@ -999,6 +999,9 @@ struct kvm_device_attr {
> > #define KVM_DEV_VFIO_GROUP1
> > #define KVM_DEV_VFIO_GROUP_ADD 1
> > #define KVM_DEV_VFIO_GROUP_DEL 2
> > +#define KVM_DEV_VFIO_DEVICE 2
> > +#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
> > +#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
> same here
> Best Regards
>
> Eric
> >
> > enum kvm_device_type {
> > KVM_DEV_TYPE_F
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
On 06/11/2015 01:51 PM, Feng Wu wrote:
From: Eric Auger
This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
to set a VFIO device IRQ as forwarded or not forwarded.
the command takes as argument a handle to a new struct named
kvm_vfio_dev_irq.
Is there no way to do this automatically? After all, vfio knows that a
device interrupt is forwarded to some eventfd, and kvm knows that some
eventfd is forwarded to a guest interrupt. If they compare notes
through a central registry, they can figure out that the interrupt needs
to be forwarded.
Signed-off-by: Eric Auger
---
Documentation/virtual/kvm/devices/vfio.txt | 34 --
include/uapi/linux/kvm.h | 12 +++
2 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/Documentation/virtual/kvm/devices/vfio.txt
b/Documentation/virtual/kvm/devices/vfio.txt
index ef51740..6186e6d 100644
--- a/Documentation/virtual/kvm/devices/vfio.txt
+++ b/Documentation/virtual/kvm/devices/vfio.txt
@@ -4,15 +4,20 @@ VFIO virtual device
Device types supported:
KVM_DEV_TYPE_VFIO
-Only one VFIO instance may be created per VM. The created device
-tracks VFIO groups in use by the VM and features of those groups
-important to the correctness and acceleration of the VM. As groups
-are enabled and disabled for use by the VM, KVM should be updated
-about their presence. When registered with KVM, a reference to the
-VFIO-group is held by KVM.
+Only one VFIO instance may be created per VM.
+
+The created device tracks VFIO groups in use by the VM and features
+of those groups important to the correctness and acceleration of
+the VM. As groups are enabled and disabled for use by the VM, KVM
+should be updated about their presence. When registered with KVM,
+a reference to the VFIO-group is held by KVM.
+
+The device also enables to control some IRQ settings of VFIO devices:
+forwarding/posting.
Groups:
KVM_DEV_VFIO_GROUP
+ KVM_DEV_VFIO_DEVICE
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
@@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes:
For each, kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
+
+KVM_DEV_VFIO_DEVICE attributes:
+ KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as forwarded
+ KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as not forwarded
+
+For each, kvm_device_attr.addr points to a kvm_vfio_dev_irq struct.
+
+When forwarded, a physical IRQ is completed by the guest and not by the
+host. This requires HW support in the interrupt controller.
+
+Forwarding can only be set when the corresponding VFIO IRQ is not masked
+(would it be through VFIO_DEVICE_SET_IRQS command or as a consequence of this
+IRQ being currently handled) or active at interrupt controller level.
+In such a situation, -EAGAIN is returned. It is advised to to set the
+forwarding before the VFIO signaling is set up, this avoids trial and errors.
+
+Unforwarding can happen at any time.
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4b60056..798f3e4 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -999,6 +999,9 @@ struct kvm_device_attr {
#define KVM_DEV_VFIO_GROUP 1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_DEVICE 2
+#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
+#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
enum kvm_device_type {
KVM_DEV_TYPE_FSL_MPIC_20= 1,
@@ -1018,6 +1021,15 @@ enum kvm_device_type {
KVM_DEV_TYPE_MAX,
};
+struct kvm_vfio_dev_irq {
+ __u32 argsz; /* structure length */
+ __u32 fd; /* file descriptor of the VFIO device */
+ __u32 index; /* VFIO device IRQ index */
+ __u32 start; /* start of subindex range */
+ __u32 count; /* size of subindex range */
+ __u32 gsi[]; /* gsi, ie. virtual IRQ number */
+};
+
/*
* ioctls for VM fds
*/
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
Hi Feng,
On 06/11/2015 12:51 PM, Feng Wu wrote:
> From: Eric Auger
>
> This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
> and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
> KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
> to set a VFIO device IRQ as forwarded or not forwarded.
> the command takes as argument a handle to a new struct named
> kvm_vfio_dev_irq.
>
> Signed-off-by: Eric Auger
> ---
> Documentation/virtual/kvm/devices/vfio.txt | 34
> --
> include/uapi/linux/kvm.h | 12 +++
> 2 files changed, 40 insertions(+), 6 deletions(-)
>
> diff --git a/Documentation/virtual/kvm/devices/vfio.txt
> b/Documentation/virtual/kvm/devices/vfio.txt
> index ef51740..6186e6d 100644
> --- a/Documentation/virtual/kvm/devices/vfio.txt
> +++ b/Documentation/virtual/kvm/devices/vfio.txt
> @@ -4,15 +4,20 @@ VFIO virtual device
> Device types supported:
>KVM_DEV_TYPE_VFIO
>
> -Only one VFIO instance may be created per VM. The created device
> -tracks VFIO groups in use by the VM and features of those groups
> -important to the correctness and acceleration of the VM. As groups
> -are enabled and disabled for use by the VM, KVM should be updated
> -about their presence. When registered with KVM, a reference to the
> -VFIO-group is held by KVM.
> +Only one VFIO instance may be created per VM.
> +
> +The created device tracks VFIO groups in use by the VM and features
> +of those groups important to the correctness and acceleration of
> +the VM. As groups are enabled and disabled for use by the VM, KVM
> +should be updated about their presence. When registered with KVM,
> +a reference to the VFIO-group is held by KVM.
> +
> +The device also enables to control some IRQ settings of VFIO devices:
> +forwarding/posting.
>
> Groups:
>KVM_DEV_VFIO_GROUP
> + KVM_DEV_VFIO_DEVICE
>
> KVM_DEV_VFIO_GROUP attributes:
>KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
> @@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes:
>
> For each, kvm_device_attr.addr points to an int32_t file descriptor
> for the VFIO group.
> +
> +KVM_DEV_VFIO_DEVICE attributes:
> + KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as forwarded
> + KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as not forwarded
> +
> +For each, kvm_device_attr.addr points to a kvm_vfio_dev_irq struct.
> +
> +When forwarded, a physical IRQ is completed by the guest and not by the
> +host. This requires HW support in the interrupt controller.
> +
> +Forwarding can only be set when the corresponding VFIO IRQ is not masked
> +(would it be through VFIO_DEVICE_SET_IRQS command or as a consequence of this
> +IRQ being currently handled) or active at interrupt controller level.
> +In such a situation, -EAGAIN is returned. It is advised to to set the
> +forwarding before the VFIO signaling is set up, this avoids trial and errors.
> +
> +Unforwarding can happen at any time.
Unfortunately the above text is out of context of your series. If it
simplifies things take the ownership of that patch file and remove my
description of KVM_DEV_VFIO_DEVICE_FORWARD_IRQ.
> diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
> index 4b60056..798f3e4 100644
> --- a/include/uapi/linux/kvm.h
> +++ b/include/uapi/linux/kvm.h
> @@ -999,6 +999,9 @@ struct kvm_device_attr {
> #define KVM_DEV_VFIO_GROUP 1
> #define KVM_DEV_VFIO_GROUP_ADD 1
> #define KVM_DEV_VFIO_GROUP_DEL 2
> +#define KVM_DEV_VFIO_DEVICE 2
> +#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ1
> +#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ 2
same here
Best Regards
Eric
>
> enum kvm_device_type {
> KVM_DEV_TYPE_FSL_MPIC_20= 1,
> @@ -1018,6 +1021,15 @@ enum kvm_device_type {
> KVM_DEV_TYPE_MAX,
> };
>
> +struct kvm_vfio_dev_irq {
> + __u32 argsz; /* structure length */
> + __u32 fd; /* file descriptor of the VFIO device */
> + __u32 index; /* VFIO device IRQ index */
> + __u32 start; /* start of subindex range */
> + __u32 count; /* size of subindex range */
> + __u32 gsi[]; /* gsi, ie. virtual IRQ number */
> +};
> +
> /*
> * ioctls for VM fds
> */
>
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
[v4 08/16] KVM: kvm-vfio: User API for IRQ forwarding
From: Eric Auger
This patch adds and documents a new KVM_DEV_VFIO_DEVICE group
and 2 device attributes: KVM_DEV_VFIO_DEVICE_FORWARD_IRQ,
KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ. The purpose is to be able
to set a VFIO device IRQ as forwarded or not forwarded.
the command takes as argument a handle to a new struct named
kvm_vfio_dev_irq.
Signed-off-by: Eric Auger
---
Documentation/virtual/kvm/devices/vfio.txt | 34 --
include/uapi/linux/kvm.h | 12 +++
2 files changed, 40 insertions(+), 6 deletions(-)
diff --git a/Documentation/virtual/kvm/devices/vfio.txt
b/Documentation/virtual/kvm/devices/vfio.txt
index ef51740..6186e6d 100644
--- a/Documentation/virtual/kvm/devices/vfio.txt
+++ b/Documentation/virtual/kvm/devices/vfio.txt
@@ -4,15 +4,20 @@ VFIO virtual device
Device types supported:
KVM_DEV_TYPE_VFIO
-Only one VFIO instance may be created per VM. The created device
-tracks VFIO groups in use by the VM and features of those groups
-important to the correctness and acceleration of the VM. As groups
-are enabled and disabled for use by the VM, KVM should be updated
-about their presence. When registered with KVM, a reference to the
-VFIO-group is held by KVM.
+Only one VFIO instance may be created per VM.
+
+The created device tracks VFIO groups in use by the VM and features
+of those groups important to the correctness and acceleration of
+the VM. As groups are enabled and disabled for use by the VM, KVM
+should be updated about their presence. When registered with KVM,
+a reference to the VFIO-group is held by KVM.
+
+The device also enables to control some IRQ settings of VFIO devices:
+forwarding/posting.
Groups:
KVM_DEV_VFIO_GROUP
+ KVM_DEV_VFIO_DEVICE
KVM_DEV_VFIO_GROUP attributes:
KVM_DEV_VFIO_GROUP_ADD: Add a VFIO group to VFIO-KVM device tracking
@@ -20,3 +25,20 @@ KVM_DEV_VFIO_GROUP attributes:
For each, kvm_device_attr.addr points to an int32_t file descriptor
for the VFIO group.
+
+KVM_DEV_VFIO_DEVICE attributes:
+ KVM_DEV_VFIO_DEVICE_FORWARD_IRQ: set a VFIO device IRQ as forwarded
+ KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ: set a VFIO device IRQ as not forwarded
+
+For each, kvm_device_attr.addr points to a kvm_vfio_dev_irq struct.
+
+When forwarded, a physical IRQ is completed by the guest and not by the
+host. This requires HW support in the interrupt controller.
+
+Forwarding can only be set when the corresponding VFIO IRQ is not masked
+(would it be through VFIO_DEVICE_SET_IRQS command or as a consequence of this
+IRQ being currently handled) or active at interrupt controller level.
+In such a situation, -EAGAIN is returned. It is advised to to set the
+forwarding before the VFIO signaling is set up, this avoids trial and errors.
+
+Unforwarding can happen at any time.
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 4b60056..798f3e4 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -999,6 +999,9 @@ struct kvm_device_attr {
#define KVM_DEV_VFIO_GROUP1
#define KVM_DEV_VFIO_GROUP_ADD 1
#define KVM_DEV_VFIO_GROUP_DEL 2
+#define KVM_DEV_VFIO_DEVICE 2
+#define KVM_DEV_VFIO_DEVICE_FORWARD_IRQ 1
+#define KVM_DEV_VFIO_DEVICE_UNFORWARD_IRQ2
enum kvm_device_type {
KVM_DEV_TYPE_FSL_MPIC_20= 1,
@@ -1018,6 +1021,15 @@ enum kvm_device_type {
KVM_DEV_TYPE_MAX,
};
+struct kvm_vfio_dev_irq {
+ __u32 argsz; /* structure length */
+ __u32 fd; /* file descriptor of the VFIO device */
+ __u32 index; /* VFIO device IRQ index */
+ __u32 start; /* start of subindex range */
+ __u32 count; /* size of subindex range */
+ __u32 gsi[]; /* gsi, ie. virtual IRQ number */
+};
+
/*
* ioctls for VM fds
*/
--
2.1.0
--
To unsubscribe from this list: send the line "unsubscribe kvm" in
the body of a message to majord...@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
