Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On Wed, Dec 16, 2015 at 10:55:12PM +0100, Paolo Bonzini wrote: > > > On 16/12/2015 20:15, Alex Williamson wrote: > > The consumers would be, for instance, Intel PI + the threaded handler > > added in this series. These run independently, the PI bypass simply > > makes the interrupt disappear from the host when it catches it, but if > > the vCPU isn't running in the right place at the time of the interrupt, > > it gets delivered to the host, in which case the secondary consumer > > implementing handle_irq() provides a lower latency injection than the > > eventfd path. If PI isn't supported, only this latter consumer is > > registered. > > I would implement the two in a single consumer, knowing that only one of > the two parts would effectively run. But because of the possibility of > multiple consumers implementing handle_irq(), I am not sure if this is > feasible. So is it possible that we limit only one consumer with handle_irq(), as my previous response to Alex? We can extend it in future if we do need support multiple consumder implementing handle_irq()? Thanks --jyh > > > On the surface it seems like a reasonable solution, though having > > multiple consumers implementing handle_irq() seems problematic. Do we > > get multiple injections if we call them all? > > Indeed. > > > Should we have some way > > to prioritize one handler versus another? Perhaps KVM should have a > > single unified consumer that can provide that sort of logic, though we > > still need the srcu code added here to protect against registration and > > irq_handler() races. Thanks, > > I'm happy to see that we have the same doubts. :) > > Paolo > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On Wed, Dec 16, 2015 at 12:15:23PM -0700, Alex Williamson wrote: > On Wed, 2015-12-16 at 18:56 +0100, Paolo Bonzini wrote: > > Alex, > > > > can you take a look at the extension to the irq bypass interface in > > patch 2? I'm not sure I understand what is the case where you have > > multiple consumers for the same token. > > The consumers would be, for instance, Intel PI + the threaded handler > added in this series. These run independently, the PI bypass simply > makes the interrupt disappear from the host when it catches it, but if > the vCPU isn't running in the right place at the time of the interrupt, > it gets delivered to the host, in which case the secondary consumer > implementing handle_irq() provides a lower latency injection than the Sorry for slow response. If the PI is delivered to the host because guest is not running, I think it will not trigger the secondary consumer. The reason is, with PI, the interrupt will be delivered as the POSTED_INTR_VECTOR or POSTED_INTR_WAKEUP_VECTOR. So for the PI consumer will not be invoked on run time scenario. > eventfd path. If PI isn't supported, only this latter consumer is > registered. > > On the surface it seems like a reasonable solution, though having > multiple consumers implementing handle_irq() seems problematic. Do we Yes, agree that has multiple consumers implementing handle_irq() seems not good. But I do think it can be helpful. A naive case is, a consumer can be created to log all the interrupt event, or to create a pipe for analysis. > get multiple injections if we call them all? Should we have some way As discussed above, currently I think we have only one consumer to handle_irq(), so it should be ok? We can limit the framework to support only one consumer with handle_irq()? > to prioritize one handler versus another? Perhaps KVM should have a > single unified consumer that can provide that sort of logic, though we I'd think still different consumer for the PI and this fast_IRQ. Thanks --jyh > still need the srcu code added here to protect against registration and > irq_handler() races. Thanks, > > Alex > > > On 03/12/2015 19:22, Yunhong Jiang wrote: > > > When assigning a VFIO device to a KVM guest with low latency > > > requirement, it > > > is better to handle the interrupt in the hard interrupt context, to > > > reduce > > > the context switch to/from the IRQ thread. > > > > > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the > > > VFIO msi > > > interrupt is changed to use request_threaded_irq(). The primary > > > interrupt > > > handler tries to set the guest interrupt atomically. If it fails to > > > achieve > > > it, a threaded interrupt handler will be invoked. > > > > > > The irq_bypass manager is extended for this purpose. The KVM > > > eventfd will > > > provide a irqbypass consumer to handle the interrupt at hard > > > interrupt > > > context. The producer will invoke the consumer's handler then. > > > > > > Yunhong Jiang (5): > > > Extract the irqfd_wakeup_pollin/irqfd_wakeup_pollup > > > Support runtime irq_bypass consumer > > > Support threaded interrupt handling on VFIO > > > Add the irq handling consumer > > > Expose x86 kvm_arch_set_irq_inatomic() > > > > > > arch/x86/kvm/Kconfig | 1 + > > > drivers/vfio/pci/vfio_pci_intrs.c | 39 ++-- > > > include/linux/irqbypass.h | 8 +++ > > > include/linux/kvm_host.h | 19 +- > > > include/linux/kvm_irqfd.h | 1 + > > > virt/kvm/Kconfig | 3 + > > > virt/kvm/eventfd.c| 131 > > > ++ > > > virt/lib/irqbypass.c | 82 ++-- > > > 8 files changed, 214 insertions(+), 70 deletions(-) > > > > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
Alex, can you take a look at the extension to the irq bypass interface in patch 2? I'm not sure I understand what is the case where you have multiple consumers for the same token. Paolo On 03/12/2015 19:22, Yunhong Jiang wrote: > When assigning a VFIO device to a KVM guest with low latency requirement, it > is better to handle the interrupt in the hard interrupt context, to reduce > the context switch to/from the IRQ thread. > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi > interrupt is changed to use request_threaded_irq(). The primary interrupt > handler tries to set the guest interrupt atomically. If it fails to achieve > it, a threaded interrupt handler will be invoked. > > The irq_bypass manager is extended for this purpose. The KVM eventfd will > provide a irqbypass consumer to handle the interrupt at hard interrupt > context. The producer will invoke the consumer's handler then. > > Yunhong Jiang (5): > Extract the irqfd_wakeup_pollin/irqfd_wakeup_pollup > Support runtime irq_bypass consumer > Support threaded interrupt handling on VFIO > Add the irq handling consumer > Expose x86 kvm_arch_set_irq_inatomic() > > arch/x86/kvm/Kconfig | 1 + > drivers/vfio/pci/vfio_pci_intrs.c | 39 ++-- > include/linux/irqbypass.h | 8 +++ > include/linux/kvm_host.h | 19 +- > include/linux/kvm_irqfd.h | 1 + > virt/kvm/Kconfig | 3 + > virt/kvm/eventfd.c| 131 > ++ > virt/lib/irqbypass.c | 82 ++-- > 8 files changed, 214 insertions(+), 70 deletions(-) > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On Wed, 2015-12-16 at 18:56 +0100, Paolo Bonzini wrote: > Alex, > > can you take a look at the extension to the irq bypass interface in > patch 2? I'm not sure I understand what is the case where you have > multiple consumers for the same token. The consumers would be, for instance, Intel PI + the threaded handler added in this series. These run independently, the PI bypass simply makes the interrupt disappear from the host when it catches it, but if the vCPU isn't running in the right place at the time of the interrupt, it gets delivered to the host, in which case the secondary consumer implementing handle_irq() provides a lower latency injection than the eventfd path. If PI isn't supported, only this latter consumer is registered. On the surface it seems like a reasonable solution, though having multiple consumers implementing handle_irq() seems problematic. Do we get multiple injections if we call them all? Should we have some way to prioritize one handler versus another? Perhaps KVM should have a single unified consumer that can provide that sort of logic, though we still need the srcu code added here to protect against registration and irq_handler() races. Thanks, Alex > On 03/12/2015 19:22, Yunhong Jiang wrote: > > When assigning a VFIO device to a KVM guest with low latency > > requirement, it > > is better to handle the interrupt in the hard interrupt context, to > > reduce > > the context switch to/from the IRQ thread. > > > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the > > VFIO msi > > interrupt is changed to use request_threaded_irq(). The primary > > interrupt > > handler tries to set the guest interrupt atomically. If it fails to > > achieve > > it, a threaded interrupt handler will be invoked. > > > > The irq_bypass manager is extended for this purpose. The KVM > > eventfd will > > provide a irqbypass consumer to handle the interrupt at hard > > interrupt > > context. The producer will invoke the consumer's handler then. > > > > Yunhong Jiang (5): > > Extract the irqfd_wakeup_pollin/irqfd_wakeup_pollup > > Support runtime irq_bypass consumer > > Support threaded interrupt handling on VFIO > > Add the irq handling consumer > > Expose x86 kvm_arch_set_irq_inatomic() > > > > arch/x86/kvm/Kconfig | 1 + > > drivers/vfio/pci/vfio_pci_intrs.c | 39 ++-- > > include/linux/irqbypass.h | 8 +++ > > include/linux/kvm_host.h | 19 +- > > include/linux/kvm_irqfd.h | 1 + > > virt/kvm/Kconfig | 3 + > > virt/kvm/eventfd.c| 131 > > ++ > > virt/lib/irqbypass.c | 82 ++-- > > 8 files changed, 214 insertions(+), 70 deletions(-) > > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On 16/12/2015 20:15, Alex Williamson wrote: > The consumers would be, for instance, Intel PI + the threaded handler > added in this series. These run independently, the PI bypass simply > makes the interrupt disappear from the host when it catches it, but if > the vCPU isn't running in the right place at the time of the interrupt, > it gets delivered to the host, in which case the secondary consumer > implementing handle_irq() provides a lower latency injection than the > eventfd path. If PI isn't supported, only this latter consumer is > registered. I would implement the two in a single consumer, knowing that only one of the two parts would effectively run. But because of the possibility of multiple consumers implementing handle_irq(), I am not sure if this is feasible. > On the surface it seems like a reasonable solution, though having > multiple consumers implementing handle_irq() seems problematic. Do we > get multiple injections if we call them all? Indeed. > Should we have some way > to prioritize one handler versus another? Perhaps KVM should have a > single unified consumer that can provide that sort of logic, though we > still need the srcu code added here to protect against registration and > irq_handler() races. Thanks, I'm happy to see that we have the same doubts. :) Paolo -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On Thu, Dec 03, 2015 at 11:55:53AM -0700, Alex Williamson wrote: > On Thu, 2015-12-03 at 10:22 -0800, Yunhong Jiang wrote: > > When assigning a VFIO device to a KVM guest with low latency requirement, > > it > > is better to handle the interrupt in the hard interrupt context, to reduce > > the context switch to/from the IRQ thread. > > > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi > > interrupt is changed to use request_threaded_irq(). The primary interrupt > > handler tries to set the guest interrupt atomically. If it fails to achieve > > it, a threaded interrupt handler will be invoked. > > > > The irq_bypass manager is extended for this purpose. The KVM eventfd will > > provide a irqbypass consumer to handle the interrupt at hard interrupt > > context. The producer will invoke the consumer's handler then. > > Do you have any performance data? Thanks, Sorry, I should include the data on the commit messages. The test: Launch a VM with a FPGA device, which triggers an interrpt every 1ms. The VM is launched on a isolated CPU with NONHZ_FULL enabled. Two data are collected. On the guest, a special program will check the latency from the time the interrupt been triggered to the time the interrupt received by guest IRS. On the host, I use the perf to collect the perf data. The performance Data with the patch: Host perf data: [root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10 [root@otcnfv02 test-tools]# perf kvm stat report Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time EXTERNAL_INTERRUPT 999798.55%99.31% 1.73us 17.07us 2.09us ( +- 0.21% ) PAUSE_INSTRUCTION127 1.25% 0.51% 0.69us 1.20us 0.84us ( +- 1.34% ) MSR_WRITE 20 0.20% 0.18% 1.62us 3.21us 1.93us ( +- 3.95% ) Guest data: [nfv@rt-test-1 ~]$ ./run-int-test.sh Latency is Min: 3.74us Max: 20.08us Avg: 4.49us No of interrupts = 74995 The performance data without the patch: Host perf data: [root@otcnfv02 test-tools]# perf kvm stat record -C 34 sleep 10 [root@otcnfv02 test-tools]# perf kvm stat report Analyze events for all VMs, all VCPUs: VM-EXITSamples Samples% Time%Min TimeMax Time Avg time PAUSE_INSTRUCTION 14113687.74%50.39% 0.69us 8.51us 0.77us ( +- 0.07% ) EXTERNAL_INTERRUPT 1970112.25%49.59% 2.31us 15.93us 5.46us ( +- 0.12% ) MSR_WRITE 24 0.01% 0.02% 1.51us 2.22us 1.91us ( +- 1.91% ) Notice: The EXTERNAL_INTERRUPT VMExit is different w/ the patch (9997) and w/o the patch (19701). It is because with threaded IRQ, the NOHZ_FULL it not working because two threads on the pCPU, thus we have both the FPGA device interrupt and the tick timer interrupt. After calculation, the average time for the FPGA device interrupt is 4.72 us. Guest data: [nfv@rt-test-1 ~]$ ./run-int-test.sh Latency is Min: 6.70us Max: 50.38us Avg: 7.44us No of interrupts = 42639 Thanks --jyh > > Alex > -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [PATCH 0/5] Threaded MSI interrupt for VFIO PCI device
On Thu, 2015-12-03 at 10:22 -0800, Yunhong Jiang wrote: > When assigning a VFIO device to a KVM guest with low latency requirement, it > is better to handle the interrupt in the hard interrupt context, to reduce > the context switch to/from the IRQ thread. > > Based on discussion on https://lkml.org/lkml/2015/10/26/764, the VFIO msi > interrupt is changed to use request_threaded_irq(). The primary interrupt > handler tries to set the guest interrupt atomically. If it fails to achieve > it, a threaded interrupt handler will be invoked. > > The irq_bypass manager is extended for this purpose. The KVM eventfd will > provide a irqbypass consumer to handle the interrupt at hard interrupt > context. The producer will invoke the consumer's handler then. Do you have any performance data? Thanks, Alex -- To unsubscribe from this list: send the line "unsubscribe kvm" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
