Re: Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-09 Thread Keith Busch
On Wed, Aug 10, 2022 at 12:48:53AM +0800, 樊金昊 wrote: >> The driver will create the cq with an allocated vector, but it's not >> activated >> until after the driver wires it up to a handler. I think that's what you're >> observing with the incomplete MSIx table entry on creation. > > Also, I'm won

Re: Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-09 Thread 樊金昊
> The driver will create the cq with an allocated vector, but it's not activated > until after the driver wires it up to a handler. I think that's what you're > observing with the incomplete MSIx table entry on creation. Also, I'm wondering if this is inconsistent with the NVMe spec. In Section 7

Re: Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-09 Thread 樊金昊
> The driver will create the cq with an allocated vector, but it's not activated > until after the driver wires it up to a handler. I think that's what you're > observing with the incomplete MSIx table entry on creation. Agreed. I digged through pci_request_irq()'s call chain and found pci_write

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-09 Thread Keith Busch
On Mon, Aug 08, 2022 at 10:23:03AM +0800, Jinhao Fan wrote: > at 12:35 PM, Jinhao Fan wrote: > > > static void nvme_irq_assert(NvmeCtrl *n, NvmeCQueue *cq) > > { > > if (cq->irq_enabled) { > > if (msix_enabled(&(n->parent_obj))) { > > +/* Initialize CQ irqfd */ > > +

Re: Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-09 Thread 樊金昊
> In this version I left irqfd initialization to the first assertion of an > irq. But I think it is better to initialize irqfd at cq creation time so we > won’t bother checking it at each irq assertion. However if I put these code > in nvme_init_cq(), irqfd does not work properly. After adding some

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-07 Thread Jinhao Fan
at 12:35 PM, Jinhao Fan wrote: > static void nvme_irq_assert(NvmeCtrl *n, NvmeCQueue *cq) > { > if (cq->irq_enabled) { > if (msix_enabled(&(n->parent_obj))) { > +/* Initialize CQ irqfd */ > +if (!cq->irqfd_enabled && n->params.ioeventfd && cq->cqid != 0) { > +

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-01 Thread Klaus Jensen
On Aug 2 12:03, Jinhao Fan wrote: > at 6:21 PM, Stefan Hajnoczi wrote: > > > What happens when the MSI-X vector is masked? > > > > I remember the VIRTIO code having masking support. I'm on my phone and > > can't check now, but I think it registers a temporary eventfd and buffers > > irqs whil

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-08-01 Thread Jinhao Fan
at 6:21 PM, Stefan Hajnoczi wrote: > What happens when the MSI-X vector is masked? > > I remember the VIRTIO code having masking support. I'm on my phone and can't > check now, but I think it registers a temporary eventfd and buffers irqs > while the vector is masked. Hi Stefan, While implem

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-28 Thread Jinhao Fan
at 11:18 PM, Stefan Hajnoczi wrote: > I think that is incorrect. QEMU has guest notifier emulation for the > non-KVM (and non-MSI-X PCI) cases. When there is no irqfd support > available, QEMU sets up a regular eventfd and calls > virtio_queue_guest_notifier_read() when it becomes readable. Than

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-28 Thread Stefan Hajnoczi
On Thu, Jul 28, 2022, 11:34 Jinhao Fan wrote: > at 11:18 PM, Stefan Hajnoczi wrote: > > > I think that is incorrect. QEMU has guest notifier emulation for the > > non-KVM (and non-MSI-X PCI) cases. When there is no irqfd support > > available, QEMU sets up a regular eventfd and calls > > virtio_

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-27 Thread Stefan Hajnoczi
On Wed, 27 Jul 2022 at 03:18, Klaus Jensen wrote: > > On Jul 21 09:29, Stefan Hajnoczi wrote: > > On Wed, Jul 20, 2022, 22:36 Jinhao Fan wrote: > > > > > Hi Stefan, > > > > > > Thanks for the detailed explanation! > > > > > > at 6:21 PM, Stefan Hajnoczi wrote: > > > > > > > Hi Jinhao, > > > > Th

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-27 Thread Klaus Jensen
On Jul 21 09:29, Stefan Hajnoczi wrote: > On Wed, Jul 20, 2022, 22:36 Jinhao Fan wrote: > > > Hi Stefan, > > > > Thanks for the detailed explanation! > > > > at 6:21 PM, Stefan Hajnoczi wrote: > > > > > Hi Jinhao, > > > Thanks for working on this! > > > > > > irqfd is not necessarily faster than

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-24 Thread Jinhao Fan
at 3:36 AM, Stefan Hajnoczi wrote: > > > On Sun, Jul 24, 2022, 11:21 Jinhao Fan wrote: > at 9:29 PM, Stefan Hajnoczi wrote: > > > > > Nice, perf(1) is good for that. You can enable trace events and add > > kprobes/uprobes to record timestamps when specific functions are entered. > > > > T

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-24 Thread Stefan Hajnoczi
On Sun, Jul 24, 2022, 11:21 Jinhao Fan wrote: > at 9:29 PM, Stefan Hajnoczi wrote: > > > > > Nice, perf(1) is good for that. You can enable trace events and add > > kprobes/uprobes to record timestamps when specific functions are entered. > > > > Thanks Stefan, > > One last question: Currently w

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-24 Thread Jinhao Fan
at 9:29 PM, Stefan Hajnoczi wrote: > > Nice, perf(1) is good for that. You can enable trace events and add > kprobes/uprobes to record timestamps when specific functions are entered. > Thanks Stefan, One last question: Currently we can achieve hundreds of KIOPS. That means perf can easily cap

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-21 Thread Stefan Hajnoczi
On Wed, Jul 20, 2022, 22:36 Jinhao Fan wrote: > Hi Stefan, > > Thanks for the detailed explanation! > > at 6:21 PM, Stefan Hajnoczi wrote: > > > Hi Jinhao, > > Thanks for working on this! > > > > irqfd is not necessarily faster than KVM ioctl interrupt injection. > > > > There are at least two n

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-20 Thread Jinhao Fan
Hi Stefan, Thanks for the detailed explanation! at 6:21 PM, Stefan Hajnoczi wrote: > Hi Jinhao, > Thanks for working on this! > > irqfd is not necessarily faster than KVM ioctl interrupt injection. > > There are at least two non performance reasons for irqfd: > 1. It avoids QEMU emulation co

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-20 Thread Stefan Hajnoczi
On Sat, Jul 9, 2022, 00:36 Jinhao Fan wrote: > Use irqfd to directly notify KVM to inject interrupts. This is done by > registering a virtual IRQ(virq) in KVM and associate the virq with an > irqfd, so that KVM can directly inject the interrupt when it receives > notification from the irqfd. This

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-20 Thread Jinhao Fan
at 12:35 PM, Jinhao Fan wrote: > Use irqfd to directly notify KVM to inject interrupts. This is done by > registering a virtual IRQ(virq) in KVM and associate the virq with an > irqfd, so that KVM can directly inject the interrupt when it receives > notification from the irqfd. This approach is s

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-14 Thread Jinhao Fan
at 12:18 PM, Klaus Jensen wrote: > On Jul 12 14:26, Klaus Jensen wrote: >> On Jul 9 12:35, Jinhao Fan wrote: >>> Use irqfd to directly notify KVM to inject interrupts. This is done by >>> registering a virtual IRQ(virq) in KVM and associate the virq with an >>> irqfd, so that KVM can directly in

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-13 Thread Klaus Jensen
On Jul 12 14:26, Klaus Jensen wrote: > On Jul 9 12:35, Jinhao Fan wrote: > > Use irqfd to directly notify KVM to inject interrupts. This is done by > > registering a virtual IRQ(virq) in KVM and associate the virq with an > > irqfd, so that KVM can directly inject the interrupt when it receives >

Re: [RFC] hw/nvme: Use irqfd to send interrupts

2022-07-12 Thread Klaus Jensen
On Jul 9 12:35, Jinhao Fan wrote: > Use irqfd to directly notify KVM to inject interrupts. This is done by > registering a virtual IRQ(virq) in KVM and associate the virq with an > irqfd, so that KVM can directly inject the interrupt when it receives > notification from the irqfd. This approach is

[RFC] hw/nvme: Use irqfd to send interrupts

2022-07-08 Thread Jinhao Fan
Use irqfd to directly notify KVM to inject interrupts. This is done by registering a virtual IRQ(virq) in KVM and associate the virq with an irqfd, so that KVM can directly inject the interrupt when it receives notification from the irqfd. This approach is supposed to improve performance because i