Re: [PULL 1/1] virtio-pci: fix use of a released vector
On Tue, Apr 16, 2024 at 8:22 PM Peter Maydell wrote:
>
> On Tue, 16 Apr 2024 at 12:50, Peter Maydell wrote:
> >
> > On Tue, 16 Apr 2024 at 12:05, Cindy Lu wrote:
> > >
> > > On Tue, Apr 16, 2024 at 6:01 PM Peter Maydell
> > > wrote:
> > > > Hi; Coverity points out what it thinks is a problem in
> > > > this commit (CID 1543938):
>
> > > > Here we pass that through to kvm_virtio_pci_vector_use_one().
> > > > In kvm_virtio_pci_vector_use_one()'s error-exit path ("undo")
> > > > it does
> > > > vector = virtio_queue_vector(vdev, queue_no);
> > > > and in virtio_queue_vector() it does:
> > > >
> > > > return n < VIRTIO_QUEUE_MAX ? vdev->vq[n].vector :
> > > > VIRTIO_NO_VECTOR;
> > > >
> > > > where 'n' is an int, so if we can get here with queue_no being
> > > > VIRTIO_CONFIG_IRQ_IDX then we'll index off the front of the
> > > > vdev->vq[] array.
> > > >
> > > > Maybe this is a "can't happen" case, but it does seem odd that
> > > > virtio_queue_vector() only bounds-checks the "too big" case
> > > > for its argument and not the "too small" case and/or it
> > > > doesn't have a special case for VIRTIO_CONFIG_IRQ_IDX.
> > > >
> > > > > +}
> > > > > +}
> > > > > +
> > > >
> > > hi peter
> > > I think we can simply remove the part
> > > vector = virtio_queue_vector(vdev, queue_no);
> > > the vector is get from virtio_pci_get_notifier() and don't need to get it
> > > again
> > > I will send the fix soon
> >
> > The error handling in kvm_virtio_pci_vector_use_one() looks
> > a bit odd in other ways, too. The only bit of "undoing"
> > it does as far as I can see is calling kvm_virtio_pci_irqfd_release(),
> > but there is no code path that gets to there where the
> > main codepath's call to kvm_virtio-pci_irqfd_use() succeeded
> > and needs to be undone. So perhaps the entire "undo" code
> > block should be deleted, and the "goto undo" lines
> > replaced by simple "return ret;" ? (The codepath
> > for "kvm_virtio_pci_irqfd_use() failed" already does the
> > "kvm_virtio_pci_vq_vector_release()" by hand there.)
>
> In any case since the error handling in kvm_virtio_pci_vector_use_one()
> isn't new in this commit (you can get the same problem via
> kvm_virtio_pci_vector_config_use(), which is CID 1468940
> first detected in 2022), I think this is not something we need
> to rush to fix before we release 9.0. If anybody disagrees now
> would be a good time to say so :-)
>
> Paolo's comment on CID 1468940 was to suggest "virtio_queue_vector
> should check VIRTIO_CONFIG_IRQ_IDX just like virtio_pci_get_notifier",
> incidentally.
>
Hi peter，
Really sorry all these mess, but I still have a stuipid question,
where can I get
this CID result ?maybe there are a mailing list？I just wonder maybe I can fix
these code earlier next time, Really thanks for your help
thanks
cindy
> thanks
> -- PMM
>
Re: [PULL 1/1] virtio-pci: fix use of a released vector
On Tue, Apr 16, 2024 at 7:50 PM Peter Maydell wrote:
>
> On Tue, 16 Apr 2024 at 12:05, Cindy Lu wrote:
> >
> > On Tue, Apr 16, 2024 at 6:01 PM Peter Maydell
> > wrote:
> > > Here we pass that through to kvm_virtio_pci_vector_use_one().
> > > In kvm_virtio_pci_vector_use_one()'s error-exit path ("undo")
> > > it does
> > > vector = virtio_queue_vector(vdev, queue_no);
> > > and in virtio_queue_vector() it does:
> > >
> > > return n < VIRTIO_QUEUE_MAX ? vdev->vq[n].vector :
> > > VIRTIO_NO_VECTOR;
> > >
> > > where 'n' is an int, so if we can get here with queue_no being
> > > VIRTIO_CONFIG_IRQ_IDX then we'll index off the front of the
> > > vdev->vq[] array.
> > >
> > > Maybe this is a "can't happen" case, but it does seem odd that
> > > virtio_queue_vector() only bounds-checks the "too big" case
> > > for its argument and not the "too small" case and/or it
> > > doesn't have a special case for VIRTIO_CONFIG_IRQ_IDX.
> > >
> > > > +}
> > > > +}
> > > > +
> > >
> > hi peter
> > I think we can simply remove the part
> > vector = virtio_queue_vector(vdev, queue_no);
> > the vector is get from virtio_pci_get_notifier() and don't need to get it
> > again
> > I will send the fix soon
>
> The error handling in kvm_virtio_pci_vector_use_one() looks
> a bit odd in other ways, too. The only bit of "undoing"
> it does as far as I can see is calling kvm_virtio_pci_irqfd_release(),
> but there is no code path that gets to there where the
> main codepath's call to kvm_virtio-pci_irqfd_use() succeeded
> and needs to be undone. So perhaps the entire "undo" code
> block should be deleted, and the "goto undo" lines
> replaced by simple "return ret;" ? (The codepath
> for "kvm_virtio_pci_irqfd_use() failed" already does the
> "kvm_virtio_pci_vq_vector_release()" by hand there.)
>
> thanks
> -- PMM
>
Hi peter,
Really thanks for your help. I have rewrite this part, the error
handling part is incorrect
Thanks
Cindy
Re: [PATCH] virtio-pci: Fix the failure process in kvm_virtio_pci_vector_use_one()
On Tue, Apr 16, 2024 at 8:30 PM Peter Maydell wrote:
>
> On Tue, 16 Apr 2024 at 13:29, Cindy Lu wrote:
> >
> > In function kvm_virtio_pci_vector_use_one(), in the undo label,
> > the function will get the vector incorrectly while using
> > VIRTIO_CONFIG_IRQ_IDX
> > To fix this, we remove this label and simplify the failure process
> >
> > Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
> > Cc: qemu-sta...@nongnu.org
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/virtio-pci.c | 19 +++
> > 1 file changed, 3 insertions(+), 16 deletions(-)
> >
> > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > index b138fa127a..565bdb0897 100644
> > --- a/hw/virtio/virtio-pci.c
> > +++ b/hw/virtio/virtio-pci.c
> > @@ -892,7 +892,7 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
> > *proxy, int queue_no)
> > }
> > ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
> > if (ret < 0) {
> > -goto undo;
> > +return ret;
> > }
> > /*
> > * If guest supports masking, set up irqfd now.
> > @@ -902,25 +902,12 @@ static int
> > kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
> > ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
> > if (ret < 0) {
> > kvm_virtio_pci_vq_vector_release(proxy, vector);
> > -goto undo;
> > +kvm_virtio_pci_irqfd_release(proxy, n, vector);
>
> Are you sure this is right? The kvm_virtio_pci_irqfd_use()
> just failed, so why do we need to call
> kvm_virtio_pci_irqfd_release() ?
>
> thanks
> -- PMM
>
This version should be correct. when kvm_virtio_pci_irqfd_use() fail
we need to call kvm_virtio_pci_vq_vector_release() and
kvm_virtio_pci_irqfd_release()
but for kvm_virtio_pci_vq_vector_use fail we can simple return,
in old version there is a error in failure process.
while the kvm_virtio_pci_vq_vector_use fail it call the
kvm_virtio_pci_irqfd_release,but at this time this is irqfd
is not using now
I have do the qtest and sanity test for this patch
Thanks
cindy
[PATCH] virtio-pci: Fix the failure process in kvm_virtio_pci_vector_use_one()
In function kvm_virtio_pci_vector_use_one(), in the undo label,
the function will get the vector incorrectly while using
VIRTIO_CONFIG_IRQ_IDX
To fix this, we remove this label and simplify the failure process
Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
Cc: qemu-sta...@nongnu.org
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 19 +++
1 file changed, 3 insertions(+), 16 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index b138fa127a..565bdb0897 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -892,7 +892,7 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
*proxy, int queue_no)
}
ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
if (ret < 0) {
-goto undo;
+return ret;
}
/*
* If guest supports masking, set up irqfd now.
@@ -902,25 +902,12 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
*proxy, int queue_no)
ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
if (ret < 0) {
kvm_virtio_pci_vq_vector_release(proxy, vector);
-goto undo;
+kvm_virtio_pci_irqfd_release(proxy, n, vector);
+return ret;
}
}
return 0;
-undo:
-
-vector = virtio_queue_vector(vdev, queue_no);
-if (vector >= msix_nr_vectors_allocated(dev)) {
-return ret;
-}
-if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = virtio_pci_get_notifier(proxy, queue_no, , );
-if (ret < 0) {
-return ret;
-}
-kvm_virtio_pci_irqfd_release(proxy, n, vector);
-}
-return ret;
}
static int kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy *proxy, int nvqs)
{
--
2.43.0
Re: [PULL 1/1] virtio-pci: fix use of a released vector
On Tue, Apr 16, 2024 at 6:01 PM Peter Maydell wrote:
>
> On Mon, 15 Apr 2024 at 11:52, Michael S. Tsirkin wrote:
> >
> > From: Cindy Lu
> >
> > During the booting process of the non-standard image, the behavior of the
> > called function in qemu is as follows:
> >
> > 1. vhost_net_stop() was triggered by guest image. This will call the
> > function
> > virtio_pci_set_guest_notifiers() with assgin= false,
> > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
> >
> > 2. virtio_reset() was triggered, this will set configure vector to
> > VIRTIO_NO_VECTOR
> >
> > 3.vhost_net_start() was called (at this time, the configure vector is
> > still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
> > assgin=true, so the irqfd for vector 0 is still not "init" during this
> > process
> >
> > 4. The system continues to boot and sets the vector back to 0. After that
> > msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet
> > the crash
> >
> > To fix the issue, we need to support changing the vector after
> > VIRTIO_CONFIG_S_DRIVER_OK is set.
> >
>
> Hi; Coverity points out what it thinks is a problem in this commit
> (CID 1543938):
>
> > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > index cb6940fc0e..cb159fd078 100644
> > --- a/hw/virtio/virtio-pci.c
> > +++ b/hw/virtio/virtio-pci.c
> > @@ -1424,6 +1424,38 @@ static int virtio_pci_add_mem_cap(VirtIOPCIProxy
> > *proxy,
> > return offset;
> > }
> >
> > +static void virtio_pci_set_vector(VirtIODevice *vdev,
> > + VirtIOPCIProxy *proxy,
> > + int queue_no, uint16_t old_vector,
> > + uint16_t new_vector)
> > +{
> > +bool kvm_irqfd = (vdev->status & VIRTIO_CONFIG_S_DRIVER_OK) &&
> > +msix_enabled(>pci_dev) && kvm_msi_via_irqfd_enabled();
> > +
> > +if (new_vector == old_vector) {
> > +return;
> > +}
> > +
> > +/*
> > + * If the device uses irqfd and the vector changes after DRIVER_OK is
> > + * set, we need to release the old vector and set up the new one.
> > + * Otherwise just need to set the new vector on the device.
> > + */
> > +if (kvm_irqfd && old_vector != VIRTIO_NO_VECTOR) {
> > +kvm_virtio_pci_vector_release_one(proxy, queue_no);
> > +}
> > +/* Set the new vector on the device. */
> > +if (queue_no == VIRTIO_CONFIG_IRQ_IDX) {
> > +vdev->config_vector = new_vector;
> > +} else {
> > +virtio_queue_set_vector(vdev, queue_no, new_vector);
> > +}
>
> Here queue_no can be VIRTIO_CONFIG_IRQ_IDX, which is -1.
>
> > +/* If the new vector changed need to set it up. */
> > +if (kvm_irqfd && new_vector != VIRTIO_NO_VECTOR) {
> > +kvm_virtio_pci_vector_use_one(proxy, queue_no);
>
> Here we pass that through to kvm_virtio_pci_vector_use_one().
> In kvm_virtio_pci_vector_use_one()'s error-exit path ("undo")
> it does
> vector = virtio_queue_vector(vdev, queue_no);
> and in virtio_queue_vector() it does:
>
> return n < VIRTIO_QUEUE_MAX ? vdev->vq[n].vector :
> VIRTIO_NO_VECTOR;
>
> where 'n' is an int, so if we can get here with queue_no being
> VIRTIO_CONFIG_IRQ_IDX then we'll index off the front of the
> vdev->vq[] array.
>
> Maybe this is a "can't happen" case, but it does seem odd that
> virtio_queue_vector() only bounds-checks the "too big" case
> for its argument and not the "too small" case and/or it
> doesn't have a special case for VIRTIO_CONFIG_IRQ_IDX.
>
> > +}
> > +}
> > +
>
hi peter
I think we can simply remove the part
vector = virtio_queue_vector(vdev, queue_no);
the vector is get from virtio_pci_get_notifier() and don't need to get it again
I will send the fix soon
thanks
cindy
> thanks
> -- PMM
>
Re: [PATCH v8] virtio-pci: fix use of a released vector
On Mon, Apr 15, 2024 at 5:34 PM Michael S. Tsirkin wrote:
>
> From: Cindy Lu
>
> During the booting process of the non-standard image, the behavior of the
> called function in qemu is as follows:
>
> 1. vhost_net_stop() was triggered by guest image. This will call the function
> virtio_pci_set_guest_notifiers() with assgin= false,
> virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
>
> 2. virtio_reset() was triggered, this will set configure vector to
> VIRTIO_NO_VECTOR
>
> 3.vhost_net_start() was called (at this time, the configure vector is
> still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
> assgin=true, so the irqfd for vector 0 is still not "init" during this process
>
> 4. The system continues to boot and sets the vector back to 0. After that
> msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet
> the crash
>
> To fix the issue, we need to support changing the vector after
> VIRTIO_CONFIG_S_DRIVER_OK is set.
>
> (gdb) bt
> 0 __pthread_kill_implementation (threadid=,
> signo=signo@entry=6, no_tid=no_tid@entry=0)
> at pthread_kill.c:44
> 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6,
> threadid=) at pthread_kill.c:78
> 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/posix/raise.c:26
> 3 0x7fc8714287f4 in __GI_abort () at abort.c:79
> 4 0x7fc87142871b in __assert_fail_base
> (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
> assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
> 5 0x7fc871437536 in __GI___assert_fail
> (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
> <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
> 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
> ../accel/kvm/kvm-all.c:1837
> 7 0x560640c98f8e in virtio_pci_one_vector_unmask
> (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
> n=0x560643c6e4c8)
> at ../hw/virtio/virtio-pci.c:1005
> 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
> vector=0, msg=...)
> at ../hw/virtio/virtio-pci.c:1070
> 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
> vector=0, is_masked=false)
> at ../hw/pci/msix.c:120
> 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00,
> vector=0, was_masked=true)
> at ../hw/pci/msix.c:140
> 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00,
> addr=12, val=0, size=4)
> at ../hw/pci/msix.c:231
> 12 0x560640f26d83 in memory_region_write_accessor
> (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
> mask=4294967295, attrs=...)
> at ../system/memory.c:497
> 13 0x560640f270a6 in access_with_adjusted_size
>
> (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
> access_size_max=4, access_fn=0x560640f26c8d ,
> mr=0x560643c66540, attrs=...) at ../system/memory.c:573
> 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
> addr=12, data=0, op=MO_32, attrs=...)
> at ../system/memory.c:1521
> 15 0x560640f37bac in flatview_write_continue
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028,
> len=4, addr1=12, l=4, mr=0x560643c66540)
> at ../system/physmem.c:2714
> 16 0x560640f37d0f in flatview_write
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028,
> len=4) at ../system/physmem.c:2756
> 17 0x560640f380bf in address_space_write
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4)
> at ../system/physmem.c:2863
> 18 0x560640f3812c in address_space_rw
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
> --Type for more, q to quit, c to continue without paging--
> 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
> ../accel/kvm/kvm-all.c:2915
> 20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
> ../accel/kvm/kvm-accel-ops.c:51
> 21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
> ../util/qemu-thread-posix.c:541
> 22 0x7fc87148cdcd in start_thread (arg=) at
> pthread_create.c:442
> 23 0x7fc871512630 in clone3 () at
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
> (gdb)
>
> MST: coding style and typo fixups
>
> Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
> Cc
Re: [PATCH v8] virtio-pci: fix use of a released vector
On Mon, Apr 15, 2024 at 5:34 PM Michael S. Tsirkin wrote:
>
> From: Cindy Lu
>
> During the booting process of the non-standard image, the behavior of the
> called function in qemu is as follows:
>
> 1. vhost_net_stop() was triggered by guest image. This will call the function
> virtio_pci_set_guest_notifiers() with assgin= false,
> virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
>
> 2. virtio_reset() was triggered, this will set configure vector to
> VIRTIO_NO_VECTOR
>
> 3.vhost_net_start() was called (at this time, the configure vector is
> still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
> assgin=true, so the irqfd for vector 0 is still not "init" during this process
>
> 4. The system continues to boot and sets the vector back to 0. After that
> msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet
> the crash
>
> To fix the issue, we need to support changing the vector after
> VIRTIO_CONFIG_S_DRIVER_OK is set.
>
> (gdb) bt
> 0 __pthread_kill_implementation (threadid=,
> signo=signo@entry=6, no_tid=no_tid@entry=0)
> at pthread_kill.c:44
> 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6,
> threadid=) at pthread_kill.c:78
> 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/posix/raise.c:26
> 3 0x7fc8714287f4 in __GI_abort () at abort.c:79
> 4 0x7fc87142871b in __assert_fail_base
> (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
> assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
> 5 0x7fc871437536 in __GI___assert_fail
> (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
> <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
> 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
> ../accel/kvm/kvm-all.c:1837
> 7 0x560640c98f8e in virtio_pci_one_vector_unmask
> (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
> n=0x560643c6e4c8)
> at ../hw/virtio/virtio-pci.c:1005
> 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
> vector=0, msg=...)
> at ../hw/virtio/virtio-pci.c:1070
> 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
> vector=0, is_masked=false)
> at ../hw/pci/msix.c:120
> 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00,
> vector=0, was_masked=true)
> at ../hw/pci/msix.c:140
> 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00,
> addr=12, val=0, size=4)
> at ../hw/pci/msix.c:231
> 12 0x560640f26d83 in memory_region_write_accessor
> (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
> mask=4294967295, attrs=...)
> at ../system/memory.c:497
> 13 0x560640f270a6 in access_with_adjusted_size
>
> (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
> access_size_max=4, access_fn=0x560640f26c8d ,
> mr=0x560643c66540, attrs=...) at ../system/memory.c:573
> 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
> addr=12, data=0, op=MO_32, attrs=...)
> at ../system/memory.c:1521
> 15 0x560640f37bac in flatview_write_continue
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028,
> len=4, addr1=12, l=4, mr=0x560643c66540)
> at ../system/physmem.c:2714
> 16 0x560640f37d0f in flatview_write
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028,
> len=4) at ../system/physmem.c:2756
> 17 0x560640f380bf in address_space_write
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4)
> at ../system/physmem.c:2863
> 18 0x560640f3812c in address_space_rw
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
> --Type for more, q to quit, c to continue without paging--
> 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
> ../accel/kvm/kvm-all.c:2915
> 20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
> ../accel/kvm/kvm-accel-ops.c:51
> 21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
> ../util/qemu-thread-posix.c:541
> 22 0x7fc87148cdcd in start_thread (arg=) at
> pthread_create.c:442
> 23 0x7fc871512630 in clone3 () at
> ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
> (gdb)
>
> MST: coding style and typo fixups
>
> Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
> Cc
Re: [PATCH v6] virtio-pci: Fix the crash that the vector was used after released.
On Mon, Apr 15, 2024 at 4:32 PM Michael S. Tsirkin wrote: > > On Fri, Apr 12, 2024 at 02:26:55PM +0800, Cindy Lu wrote: > > During the booting process of the non-standard image, the behavior of the > > called function in qemu is as follows: > > > > 1. vhost_net_stop() was triggered by guest image. This will call the > > function > > virtio_pci_set_guest_notifiers() with assgin= false, > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0 > > > > 2. virtio_reset() was triggered, this will set configure vector to > > VIRTIO_NO_VECTOR > > > > 3.vhost_net_start() was called (at this time, the configure vector is > > still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with > > assgin=true, so the irqfd for vector 0 is still not "init" during this > > process > > > > 4. The system continues to boot and sets the vector back to 0. After that > > msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet > > the crash > > > > To fix the issue, we need to support changing the vector after > > VIRTIO_CONFIG_S_DRIVER_OK is set. > > > > (gdb) bt > > 0 __pthread_kill_implementation (threadid=, > > signo=signo@entry=6, no_tid=no_tid@entry=0) > > at pthread_kill.c:44 > > 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, > > threadid=) at pthread_kill.c:78 > > 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at > > ../sysdeps/posix/raise.c:26 > > 3 0x7fc8714287f4 in __GI_abort () at abort.c:79 > > 4 0x7fc87142871b in __assert_fail_base > > (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", > > assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=) at > > assert.c:92 > > 5 0x7fc871437536 in __GI___assert_fail > > (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0 > > <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101 > > 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at > > ../accel/kvm/kvm-all.c:1837 > > 7 0x560640c98f8e in virtio_pci_one_vector_unmask > > (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=..., > > n=0x560643c6e4c8) > > at ../hw/virtio/virtio-pci.c:1005 > > 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00, > > vector=0, msg=...) > > at ../hw/virtio/virtio-pci.c:1070 > > 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00, > > vector=0, is_masked=false) > > at ../hw/pci/msix.c:120 > > 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, > > vector=0, was_masked=true) > > at ../hw/pci/msix.c:140 > > 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, > > addr=12, val=0, size=4) > > at ../hw/pci/msix.c:231 > > 12 0x560640f26d83 in memory_region_write_accessor > > (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0, > > mask=4294967295, attrs=...) > > at ../system/memory.c:497 > > 13 0x560640f270a6 in access_with_adjusted_size > > > > (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1, > > access_size_max=4, access_fn=0x560640f26c8d , > > mr=0x560643c66540, attrs=...) at ../system/memory.c:573 > > 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540, > > addr=12, data=0, op=MO_32, attrs=...) > > at ../system/memory.c:1521 > > 15 0x560640f37bac in flatview_write_continue > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, > > len=4, addr1=12, l=4, mr=0x560643c66540) > > at ../system/physmem.c:2714 > > 16 0x560640f37d0f in flatview_write > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, > > len=4) at ../system/physmem.c:2756 > > 17 0x560640f380bf in address_space_write > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4) > > at ../system/physmem.c:2863 > > 18 0x560640f3812c in address_space_rw > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873 > > --Type for more, q to quit, c to continue without paging-- > > 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at > > ../accel/kvm/kvm-all.c:2915 > > 20 0x560640f8d731 in kvm_vcpu_thr
Re: [PATCH v6] virtio-pci: Fix the crash that the vector was used after released.
Hi All
I apologize for bothering you again
I send the new patch is because I found that the function
kvm_virtio_pci_vector_use_one/kvm_virtio_pci_vector_release_one
can only change the vector that already set to the device.
ret = virtio_pci_get_notifier(proxy, queue_no, , );
if (ret < 0) {
return;
}
...
So I move the setting vector into the function
virtio_pci_set_and_change_vector()
the other part are the same .
the sanity test is passed and the qemu qtest is also passed
Thanks
Cindy
On Fri, Apr 12, 2024 at 2:28 PM Cindy Lu wrote:
>
> During the booting process of the non-standard image, the behavior of the
> called function in qemu is as follows:
>
> 1. vhost_net_stop() was triggered by guest image. This will call the function
> virtio_pci_set_guest_notifiers() with assgin= false,
> virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
>
> 2. virtio_reset() was triggered, this will set configure vector to
> VIRTIO_NO_VECTOR
>
> 3.vhost_net_start() was called (at this time, the configure vector is
> still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
> assgin=true, so the irqfd for vector 0 is still not "init" during this process
>
> 4. The system continues to boot and sets the vector back to 0. After that
> msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet
> the crash
>
> To fix the issue, we need to support changing the vector after
> VIRTIO_CONFIG_S_DRIVER_OK is set.
>
> (gdb) bt
> 0 __pthread_kill_implementation (threadid=,
> signo=signo@entry=6, no_tid=no_tid@entry=0)
> at pthread_kill.c:44
> 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6,
> threadid=) at pthread_kill.c:78
> 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
> ../sysdeps/posix/raise.c:26
> 3 0x7fc8714287f4 in __GI_abort () at abort.c:79
> 4 0x7fc87142871b in __assert_fail_base
> (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
> assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
> 5 0x7fc871437536 in __GI___assert_fail
> (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
> "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
> <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
> 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
> ../accel/kvm/kvm-all.c:1837
> 7 0x560640c98f8e in virtio_pci_one_vector_unmask
> (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
> n=0x560643c6e4c8)
> at ../hw/virtio/virtio-pci.c:1005
> 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
> vector=0, msg=...)
> at ../hw/virtio/virtio-pci.c:1070
> 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
> vector=0, is_masked=false)
> at ../hw/pci/msix.c:120
> 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00,
> vector=0, was_masked=true)
> at ../hw/pci/msix.c:140
> 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00,
> addr=12, val=0, size=4)
> at ../hw/pci/msix.c:231
> 12 0x560640f26d83 in memory_region_write_accessor
> (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
> mask=4294967295, attrs=...)
> at ../system/memory.c:497
> 13 0x560640f270a6 in access_with_adjusted_size
>
> (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
> access_size_max=4, access_fn=0x560640f26c8d ,
> mr=0x560643c66540, attrs=...) at ../system/memory.c:573
> 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
> addr=12, data=0, op=MO_32, attrs=...)
> at ../system/memory.c:1521
> 15 0x560640f37bac in flatview_write_continue
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028,
> len=4, addr1=12, l=4, mr=0x560643c66540)
> at ../system/physmem.c:2714
> 16 0x560640f37d0f in flatview_write
> (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028,
> len=4) at ../system/physmem.c:2756
> 17 0x560640f380bf in address_space_write
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4)
> at ../system/physmem.c:2863
> 18 0x560640f3812c in address_space_rw
> (as=0x560642161ae0 , addr=4273803276, attrs=...,
> buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
> --Type for more, q to quit, c to continue without paging--
> 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
> ../accel/kvm/kvm-all.c:2915
> 20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f20
[PATCH v6] virtio-pci: Fix the crash that the vector was used after released.
During the booting process of the non-standard image, the behavior of the
called function in qemu is as follows:
1. vhost_net_stop() was triggered by guest image. This will call the function
virtio_pci_set_guest_notifiers() with assgin= false,
virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
2. virtio_reset() was triggered, this will set configure vector to
VIRTIO_NO_VECTOR
3.vhost_net_start() was called (at this time, the configure vector is
still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
assgin=true, so the irqfd for vector 0 is still not "init" during this process
4. The system continues to boot and sets the vector back to 0. After that
msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet the
crash
To fix the issue, we need to support changing the vector after
VIRTIO_CONFIG_S_DRIVER_OK is set.
(gdb) bt
0 __pthread_kill_implementation (threadid=,
signo=signo@entry=6, no_tid=no_tid@entry=0)
at pthread_kill.c:44
1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, threadid=) at pthread_kill.c:78
2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
3 0x7fc8714287f4 in __GI_abort () at abort.c:79
4 0x7fc87142871b in __assert_fail_base
(fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
5 0x7fc871437536 in __GI___assert_fail
(assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
<__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
../accel/kvm/kvm-all.c:1837
7 0x560640c98f8e in virtio_pci_one_vector_unmask
(proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
n=0x560643c6e4c8)
at ../hw/virtio/virtio-pci.c:1005
8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
vector=0, msg=...)
at ../hw/virtio/virtio-pci.c:1070
9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
vector=0, is_masked=false)
at ../hw/pci/msix.c:120
10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, vector=0,
was_masked=true)
at ../hw/pci/msix.c:140
11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, addr=12,
val=0, size=4)
at ../hw/pci/msix.c:231
12 0x560640f26d83 in memory_region_write_accessor
(mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
mask=4294967295, attrs=...)
at ../system/memory.c:497
13 0x560640f270a6 in access_with_adjusted_size
(addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
access_size_max=4, access_fn=0x560640f26c8d ,
mr=0x560643c66540, attrs=...) at ../system/memory.c:573
14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
addr=12, data=0, op=MO_32, attrs=...)
at ../system/memory.c:1521
15 0x560640f37bac in flatview_write_continue
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, len=4,
addr1=12, l=4, mr=0x560643c66540)
at ../system/physmem.c:2714
16 0x560640f37d0f in flatview_write
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2756
17 0x560640f380bf in address_space_write
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2863
18 0x560640f3812c in address_space_rw
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
--Type for more, q to quit, c to continue without paging--
19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
../accel/kvm/kvm-all.c:2915
20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
../accel/kvm/kvm-accel-ops.c:51
21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
../util/qemu-thread-posix.c:541
22 0x7fc87148cdcd in start_thread (arg=) at
pthread_create.c:442
23 0x7fc871512630 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
Cc: qemu-sta...@nongnu.org
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 43 --
1 file changed, 41 insertions(+), 2 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 1a7039fb0c..f83ec92990 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1423,6 +1423,36 @@ static int virtio_pci_add_mem_cap(VirtIOPCIProxy *proxy,
return offset;
}
+static void virtio_pci_set_and_change_vector(VirtIODevice *vdev,
+ VirtIOPCIProxy *
[PATCH v5] virtio-pci: Fix the crash that the vector was used after released.
During the booting process of the non-standard image, the behavior of the
called function in qemu is as follows:
1. vhost_net_stop() was triggered by guest image. This will call the function
virtio_pci_set_guest_notifiers() with assgin= false,
virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
2. virtio_reset() was triggered, this will set configure vector to
VIRTIO_NO_VECTOR
3.vhost_net_start() was called (at this time, the configure vector is
still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
assgin=true, so the irqfd for vector 0 is still not "init" during this process
4. The system continues to boot and sets the vector back to 0. After that
msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet the
crash
To fix the issue, we need to support changing the vector after
VIRTIO_CONFIG_S_DRIVER_OK is set.
(gdb) bt
0 __pthread_kill_implementation (threadid=,
signo=signo@entry=6, no_tid=no_tid@entry=0)
at pthread_kill.c:44
1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, threadid=) at pthread_kill.c:78
2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
3 0x7fc8714287f4 in __GI_abort () at abort.c:79
4 0x7fc87142871b in __assert_fail_base
(fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
5 0x7fc871437536 in __GI___assert_fail
(assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
<__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
../accel/kvm/kvm-all.c:1837
7 0x560640c98f8e in virtio_pci_one_vector_unmask
(proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
n=0x560643c6e4c8)
at ../hw/virtio/virtio-pci.c:1005
8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
vector=0, msg=...)
at ../hw/virtio/virtio-pci.c:1070
9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
vector=0, is_masked=false)
at ../hw/pci/msix.c:120
10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, vector=0,
was_masked=true)
at ../hw/pci/msix.c:140
11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, addr=12,
val=0, size=4)
at ../hw/pci/msix.c:231
12 0x560640f26d83 in memory_region_write_accessor
(mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
mask=4294967295, attrs=...)
at ../system/memory.c:497
13 0x560640f270a6 in access_with_adjusted_size
(addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
access_size_max=4, access_fn=0x560640f26c8d ,
mr=0x560643c66540, attrs=...) at ../system/memory.c:573
14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
addr=12, data=0, op=MO_32, attrs=...)
at ../system/memory.c:1521
15 0x560640f37bac in flatview_write_continue
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, len=4,
addr1=12, l=4, mr=0x560643c66540)
at ../system/physmem.c:2714
16 0x560640f37d0f in flatview_write
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2756
17 0x560640f380bf in address_space_write
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2863
18 0x560640f3812c in address_space_rw
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
--Type for more, q to quit, c to continue without paging--
19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
../accel/kvm/kvm-all.c:2915
20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
../accel/kvm/kvm-accel-ops.c:51
21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
../util/qemu-thread-posix.c:541
22 0x7fc87148cdcd in start_thread (arg=) at
pthread_create.c:442
23 0x7fc871512630 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
Fixes: f9a09ca3ea ("vhost: add support for configure interrupt")
Cc: qemu-sta...@nongnu.org
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 29 +
1 file changed, 29 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 1a7039fb0c..4f2663b27d 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1423,6 +1423,24 @@ static int virtio_pci_add_mem_cap(VirtIOPCIProxy *proxy,
return offset;
}
+static void virtio_pci_vector_change(VirtIODevice *vdev, VirtIOPCIProxy *proxy,
+ int queue_no, uint16_t old_vector,
+ uint16_
Re: [PATCH v4] virtio-pci: Fix the crash that the vector was used after released.
On Thu, Apr 11, 2024 at 1:56 PM Michael S. Tsirkin wrote: > > On Thu, Apr 11, 2024 at 12:11:30PM +0800, Cindy Lu wrote: > > During the booting process of the non-standard image, the behavior of the > > called function in qemu is as follows: > > > > 1. vhost_net_stop() was triggered by guest image. This will call the > > function > > virtio_pci_set_guest_notifiers() with assgin= false, > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0 > > > > 2. virtio_reset() was triggered, this will set configure vector to > > VIRTIO_NO_VECTOR > > > > 3.vhost_net_start() was called (at this time, the configure vector is > > still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with > > assgin=true, so the irqfd for vector 0 is still not "init" during this > > process > > > > 4. The system continues to boot and sets the vector back to 0. After that > > msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet > > the crash > > > > To fix the issue, we need to support changing the vector after > > VIRTIO_CONFIG_S_DRIVER_OK is set. > > > > (gdb) bt > > 0 __pthread_kill_implementation (threadid=, > > signo=signo@entry=6, no_tid=no_tid@entry=0) > > at pthread_kill.c:44 > > 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, > > threadid=) at pthread_kill.c:78 > > 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at > > ../sysdeps/posix/raise.c:26 > > 3 0x7fc8714287f4 in __GI_abort () at abort.c:79 > > 4 0x7fc87142871b in __assert_fail_base > > (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", > > assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=) at > > assert.c:92 > > 5 0x7fc871437536 in __GI___assert_fail > > (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0 > > <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101 > > 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at > > ../accel/kvm/kvm-all.c:1837 > > 7 0x560640c98f8e in virtio_pci_one_vector_unmask > > (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=..., > > n=0x560643c6e4c8) > > at ../hw/virtio/virtio-pci.c:1005 > > 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00, > > vector=0, msg=...) > > at ../hw/virtio/virtio-pci.c:1070 > > 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00, > > vector=0, is_masked=false) > > at ../hw/pci/msix.c:120 > > 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, > > vector=0, was_masked=true) > > at ../hw/pci/msix.c:140 > > 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, > > addr=12, val=0, size=4) > > at ../hw/pci/msix.c:231 > > 12 0x560640f26d83 in memory_region_write_accessor > > (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0, > > mask=4294967295, attrs=...) > > at ../system/memory.c:497 > > 13 0x560640f270a6 in access_with_adjusted_size > > > > (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1, > > access_size_max=4, access_fn=0x560640f26c8d , > > mr=0x560643c66540, attrs=...) at ../system/memory.c:573 > > 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540, > > addr=12, data=0, op=MO_32, attrs=...) > > at ../system/memory.c:1521 > > 15 0x560640f37bac in flatview_write_continue > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, > > len=4, addr1=12, l=4, mr=0x560643c66540) > > at ../system/physmem.c:2714 > > 16 0x560640f37d0f in flatview_write > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, > > len=4) at ../system/physmem.c:2756 > > 17 0x560640f380bf in address_space_write > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4) > > at ../system/physmem.c:2863 > > 18 0x560640f3812c in address_space_rw > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873 > > --Type for more, q to quit, c to continue without paging-- > > 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at > > ../accel/kvm/kvm-all.c:2915 > > 20 0x560640f8d731 in kvm_vcpu
[PATCH v4] virtio-pci: Fix the crash that the vector was used after released.
During the booting process of the non-standard image, the behavior of the
called function in qemu is as follows:
1. vhost_net_stop() was triggered by guest image. This will call the function
virtio_pci_set_guest_notifiers() with assgin= false,
virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
2. virtio_reset() was triggered, this will set configure vector to
VIRTIO_NO_VECTOR
3.vhost_net_start() was called (at this time, the configure vector is
still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
assgin=true, so the irqfd for vector 0 is still not "init" during this process
4. The system continues to boot and sets the vector back to 0. After that
msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet the
crash
To fix the issue, we need to support changing the vector after
VIRTIO_CONFIG_S_DRIVER_OK is set.
(gdb) bt
0 __pthread_kill_implementation (threadid=,
signo=signo@entry=6, no_tid=no_tid@entry=0)
at pthread_kill.c:44
1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, threadid=) at pthread_kill.c:78
2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
3 0x7fc8714287f4 in __GI_abort () at abort.c:79
4 0x7fc87142871b in __assert_fail_base
(fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
5 0x7fc871437536 in __GI___assert_fail
(assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
<__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
../accel/kvm/kvm-all.c:1837
7 0x560640c98f8e in virtio_pci_one_vector_unmask
(proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
n=0x560643c6e4c8)
at ../hw/virtio/virtio-pci.c:1005
8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
vector=0, msg=...)
at ../hw/virtio/virtio-pci.c:1070
9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
vector=0, is_masked=false)
at ../hw/pci/msix.c:120
10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, vector=0,
was_masked=true)
at ../hw/pci/msix.c:140
11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, addr=12,
val=0, size=4)
at ../hw/pci/msix.c:231
12 0x560640f26d83 in memory_region_write_accessor
(mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
mask=4294967295, attrs=...)
at ../system/memory.c:497
13 0x560640f270a6 in access_with_adjusted_size
(addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
access_size_max=4, access_fn=0x560640f26c8d ,
mr=0x560643c66540, attrs=...) at ../system/memory.c:573
14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
addr=12, data=0, op=MO_32, attrs=...)
at ../system/memory.c:1521
15 0x560640f37bac in flatview_write_continue
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, len=4,
addr1=12, l=4, mr=0x560643c66540)
at ../system/physmem.c:2714
16 0x560640f37d0f in flatview_write
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2756
17 0x560640f380bf in address_space_write
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2863
18 0x560640f3812c in address_space_rw
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
--Type for more, q to quit, c to continue without paging--
19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
../accel/kvm/kvm-all.c:2915
20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
../accel/kvm/kvm-accel-ops.c:51
21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
../util/qemu-thread-posix.c:541
22 0x7fc87148cdcd in start_thread (arg=) at
pthread_create.c:442
23 0x7fc871512630 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 30 ++
1 file changed, 30 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 1a7039fb0c..f124a9e20b 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1570,7 +1570,22 @@ static void virtio_pci_common_write(void *opaque, hwaddr
addr,
} else {
val = VIRTIO_NO_VECTOR;
}
+vector = vdev->config_vector;
vdev->config_vector = val;
+/*
+ * If the device uses irqfd and the vector changes after DRIVER_OK is
+ * set, we need to release the old vector and set u
Re: [PATCH v3] virtio-pci: Fix the crash that the vector was used after released.
On Thu, Apr 11, 2024 at 12:18 AM Michael S. Tsirkin wrote: > > On Thu, Apr 11, 2024 at 12:12:00AM +0800, Cindy Lu wrote: > > During the booting process of the non-standard image, the behavior of the > > called function in qemu is as follows: > > > > 1. vhost_net_stop() was triggered by guest image. This will call the > > function > > virtio_pci_set_guest_notifiers() with assgin= false, > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0 > > > > 2. virtio_reset() was triggered, this will set configure vector to > > VIRTIO_NO_VECTOR > > > > 3.vhost_net_start() was called (at this time, the configure vector is > > still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with > > assgin=true, so the irqfd for vector 0 is still not "init" during this > > process > > > > 4. The system continues to boot and sets the vector back to 0. After that > > msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet > > the crash > > > > To fix the issue, we need to support changing the vector after > > VIRTIO_CONFIG_S_DRIVER_OK is set. > > > > (gdb) bt > > 0 __pthread_kill_implementation (threadid=, > > signo=signo@entry=6, no_tid=no_tid@entry=0) > > at pthread_kill.c:44 > > 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, > > threadid=) at pthread_kill.c:78 > > 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at > > ../sysdeps/posix/raise.c:26 > > 3 0x7fc8714287f4 in __GI_abort () at abort.c:79 > > 4 0x7fc87142871b in __assert_fail_base > > (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", > > assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=) at > > assert.c:92 > > 5 0x7fc871437536 in __GI___assert_fail > > (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0 > > <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101 > > 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at > > ../accel/kvm/kvm-all.c:1837 > > 7 0x560640c98f8e in virtio_pci_one_vector_unmask > > (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=..., > > n=0x560643c6e4c8) > > at ../hw/virtio/virtio-pci.c:1005 > > 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00, > > vector=0, msg=...) > > at ../hw/virtio/virtio-pci.c:1070 > > 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00, > > vector=0, is_masked=false) > > at ../hw/pci/msix.c:120 > > 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, > > vector=0, was_masked=true) > > at ../hw/pci/msix.c:140 > > 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, > > addr=12, val=0, size=4) > > at ../hw/pci/msix.c:231 > > 12 0x560640f26d83 in memory_region_write_accessor > > (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0, > > mask=4294967295, attrs=...) > > at ../system/memory.c:497 > > 13 0x560640f270a6 in access_with_adjusted_size > > > > (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1, > > access_size_max=4, access_fn=0x560640f26c8d , > > mr=0x560643c66540, attrs=...) at ../system/memory.c:573 > > 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540, > > addr=12, data=0, op=MO_32, attrs=...) > > at ../system/memory.c:1521 > > 15 0x560640f37bac in flatview_write_continue > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, > > len=4, addr1=12, l=4, mr=0x560643c66540) > > at ../system/physmem.c:2714 > > 16 0x560640f37d0f in flatview_write > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, > > len=4) at ../system/physmem.c:2756 > > 17 0x560640f380bf in address_space_write > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4) > > at ../system/physmem.c:2863 > > 18 0x560640f3812c in address_space_rw > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873 > > --Type for more, q to quit, c to continue without paging-- > > 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at > > ../accel/kvm/kvm-all.c:2915 > > 20 0x560640f8d731 in k
[PATCH v3] virtio-pci: Fix the crash that the vector was used after released.
During the booting process of the non-standard image, the behavior of the
called function in qemu is as follows:
1. vhost_net_stop() was triggered by guest image. This will call the function
virtio_pci_set_guest_notifiers() with assgin= false,
virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
2. virtio_reset() was triggered, this will set configure vector to
VIRTIO_NO_VECTOR
3.vhost_net_start() was called (at this time, the configure vector is
still VIRTIO_NO_VECTOR) and then call virtio_pci_set_guest_notifiers() with
assgin=true, so the irqfd for vector 0 is still not "init" during this process
4. The system continues to boot and sets the vector back to 0. After that
msix_fire_vector_notifier() was triggered to unmask the vector 0 and meet the
crash
To fix the issue, we need to support changing the vector after
VIRTIO_CONFIG_S_DRIVER_OK is set.
(gdb) bt
0 __pthread_kill_implementation (threadid=,
signo=signo@entry=6, no_tid=no_tid@entry=0)
at pthread_kill.c:44
1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, threadid=) at pthread_kill.c:78
2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
3 0x7fc8714287f4 in __GI_abort () at abort.c:79
4 0x7fc87142871b in __assert_fail_base
(fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
5 0x7fc871437536 in __GI___assert_fail
(assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
<__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
../accel/kvm/kvm-all.c:1837
7 0x560640c98f8e in virtio_pci_one_vector_unmask
(proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
n=0x560643c6e4c8)
at ../hw/virtio/virtio-pci.c:1005
8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
vector=0, msg=...)
at ../hw/virtio/virtio-pci.c:1070
9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
vector=0, is_masked=false)
at ../hw/pci/msix.c:120
10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, vector=0,
was_masked=true)
at ../hw/pci/msix.c:140
11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, addr=12,
val=0, size=4)
at ../hw/pci/msix.c:231
12 0x560640f26d83 in memory_region_write_accessor
(mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
mask=4294967295, attrs=...)
at ../system/memory.c:497
13 0x560640f270a6 in access_with_adjusted_size
(addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
access_size_max=4, access_fn=0x560640f26c8d ,
mr=0x560643c66540, attrs=...) at ../system/memory.c:573
14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
addr=12, data=0, op=MO_32, attrs=...)
at ../system/memory.c:1521
15 0x560640f37bac in flatview_write_continue
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, len=4,
addr1=12, l=4, mr=0x560643c66540)
at ../system/physmem.c:2714
16 0x560640f37d0f in flatview_write
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2756
17 0x560640f380bf in address_space_write
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2863
18 0x560640f3812c in address_space_rw
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
--Type for more, q to quit, c to continue without paging--
19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
../accel/kvm/kvm-all.c:2915
20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
../accel/kvm/kvm-accel-ops.c:51
21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
../util/qemu-thread-posix.c:541
22 0x7fc87148cdcd in start_thread (arg=) at
pthread_create.c:442
23 0x7fc871512630 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 30 ++
1 file changed, 30 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 1a7039fb0c..b3b1a4a66f 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1570,7 +1570,22 @@ static void virtio_pci_common_write(void *opaque, hwaddr
addr,
} else {
val = VIRTIO_NO_VECTOR;
}
+vector = vdev->config_vector;
vdev->config_vector = val;
+/*
+ * If the value was changed after DRIVER_OK was set, it means that
+ * we need to release the old vector and set up the new vector.
+
Re: [PATCH v2 1/1] virtio-pci: Fix the crash that the vector was used after released.
On Wed, Apr 10, 2024 at 1:48 PM Jason Wang wrote: > > On Wed, Apr 10, 2024 at 1:29 PM Cindy Lu wrote: > > > > When the guest triggers vhost_stop and then virtio_reset, the vector will > > the > > IRQFD for this vector will be released and change to VIRTIO_NO_VECTOR. > > After that, the guest called vhost_net_start, (at this time, the configure > > vector is still VIRTIO_NO_VECTOR), vector 0 still was not "init". > > The guest system continued to boot, set the vector back to 0, and then met > > the crash. > > Btw, the description of the cover letter seems to be better, how about > just using that (so there won't be a cover letter since this series > just have 1 patch)? > > Thanks > sure will send a new version Thanks Cindy
Re: [PATCH v2 1/1] virtio-pci: Fix the crash that the vector was used after released.
On Wed, Apr 10, 2024 at 1:36 PM Jason Wang wrote: > > On Wed, Apr 10, 2024 at 1:29 PM Cindy Lu wrote: > > > > When the guest triggers vhost_stop and then virtio_reset, the vector will > > the > > IRQFD for this vector will be released and change to VIRTIO_NO_VECTOR. > > After that, the guest called vhost_net_start, (at this time, the configure > > vector is still VIRTIO_NO_VECTOR), vector 0 still was not "init". > > The guest system continued to boot, set the vector back to 0, and then met > > the crash. > > > > To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()" > > when the vector changes back from VIRTIO_NO_VECTOR > > > > (gdb) bt > > 0 __pthread_kill_implementation (threadid=, > > signo=signo@entry=6, no_tid=no_tid@entry=0) > > at pthread_kill.c:44 > > 1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, > > threadid=) at pthread_kill.c:78 > > 2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at > > ../sysdeps/posix/raise.c:26 > > 3 0x7fc8714287f4 in __GI_abort () at abort.c:79 > > 4 0x7fc87142871b in __assert_fail_base > > (fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", > > assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=) at > > assert.c:92 > > 5 0x7fc871437536 in __GI___assert_fail > > (assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d > > "../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0 > > <__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101 > > 6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at > > ../accel/kvm/kvm-all.c:1837 > > 7 0x560640c98f8e in virtio_pci_one_vector_unmask > > (proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=..., > > n=0x560643c6e4c8) > > at ../hw/virtio/virtio-pci.c:1005 > > 8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00, > > vector=0, msg=...) > > at ../hw/virtio/virtio-pci.c:1070 > > 9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00, > > vector=0, is_masked=false) > > at ../hw/pci/msix.c:120 > > 10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, > > vector=0, was_masked=true) > > at ../hw/pci/msix.c:140 > > 11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, > > addr=12, val=0, size=4) > > at ../hw/pci/msix.c:231 > > 12 0x560640f26d83 in memory_region_write_accessor > > (mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0, > > mask=4294967295, attrs=...) > > at ../system/memory.c:497 > > 13 0x560640f270a6 in access_with_adjusted_size > > > > (addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1, > > access_size_max=4, access_fn=0x560640f26c8d , > > mr=0x560643c66540, attrs=...) at ../system/memory.c:573 > > 14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540, > > addr=12, data=0, op=MO_32, attrs=...) > > at ../system/memory.c:1521 > > 15 0x560640f37bac in flatview_write_continue > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, > > len=4, addr1=12, l=4, mr=0x560643c66540) > > at ../system/physmem.c:2714 > > 16 0x560640f37d0f in flatview_write > > (fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, > > len=4) at ../system/physmem.c:2756 > > 17 0x560640f380bf in address_space_write > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4) > > at ../system/physmem.c:2863 > > 18 0x560640f3812c in address_space_rw > > (as=0x560642161ae0 , addr=4273803276, attrs=..., > > buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873 > > --Type for more, q to quit, c to continue without paging-- > > 19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at > > ../accel/kvm/kvm-all.c:2915 > > 20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at > > ../accel/kvm/kvm-accel-ops.c:51 > > 21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at > > ../util/qemu-thread-posix.c:541 > > 22 0x7fc87148cdcd in start_thread (arg=) at > > pthread_create.c:442 > > 23 0x7fc871512630 in clone3 () at > > ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 > > (gdb) > > Signed-off-by: Cindy Lu > > --- > > hw/virtio/virtio-pci.c | 3
[PATCH v2 0/1] virtio-pci: Fix the crash that the vector was used after released
During the booting process of the Vyatta image, the behavior of the called function in qemu is as follows: 1. vhost_net_stop() was triggered by guest image . This will call the function virtio_pci_set_guest_notifiers() with assgin= false, and virtio_pci_set_guest_notifiers(??? will release the irqfd for vector 0 2. virtio_reset() was called -->set configure vector to VIRTIO_NO_VECTOR 3.vhost_net_start() was called (at this time, the configure vector is still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers() with assgin= true, so the irqfd for vector 0 is still not "init" during this process 4. The system continues to boot,set the vector back to 0, and msix_fire_vector_notifier() was triggered unmask the vector 0 and then met the crash [msix_fire_vector_notifier] 112 called vector 0 is_masked 1 [msix_fire_vector_notifier] 112 called vector 0 is_masked 0 To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()" when the vector changes back from VIRTIO_NO_VECTOR. The reason that we don't need to call kvm_virtio_pci_vector_release_one while the vector changes to VIRTIO_NO_VECTOR is this function will called in vhost_net_stop(), So this step will not lost during this process. Change from V1 1.add the check for if using irqfd 2.remove the check for bool recovery, irqfd's user is enough to check status Cindy Lu (1): virtio-pci: Fix the crash that the vector was used after released. hw/virtio/virtio-pci.c | 35 +++ 1 file changed, 35 insertions(+) -- 2.43.0
[PATCH v2 1/1] virtio-pci: Fix the crash that the vector was used after released.
When the guest triggers vhost_stop and then virtio_reset, the vector will the
IRQFD for this vector will be released and change to VIRTIO_NO_VECTOR.
After that, the guest called vhost_net_start, (at this time, the configure
vector is still VIRTIO_NO_VECTOR), vector 0 still was not "init".
The guest system continued to boot, set the vector back to 0, and then met the
crash.
To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()"
when the vector changes back from VIRTIO_NO_VECTOR
(gdb) bt
0 __pthread_kill_implementation (threadid=,
signo=signo@entry=6, no_tid=no_tid@entry=0)
at pthread_kill.c:44
1 0x7fc87148ec53 in __pthread_kill_internal (signo=6, threadid=) at pthread_kill.c:78
2 0x7fc87143e956 in __GI_raise (sig=sig@entry=6) at
../sysdeps/posix/raise.c:26
3 0x7fc8714287f4 in __GI_abort () at abort.c:79
4 0x7fc87142871b in __assert_fail_base
(fmt=0x7fc8715bbde0 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n",
assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=) at assert.c:92
5 0x7fc871437536 in __GI___assert_fail
(assertion=0x5606413efd53 "ret == 0", file=0x5606413ef87d
"../accel/kvm/kvm-all.c", line=1837, function=0x5606413f06f0
<__PRETTY_FUNCTION__.19> "kvm_irqchip_commit_routes") at assert.c:101
6 0x560640f884b5 in kvm_irqchip_commit_routes (s=0x560642cae1f0) at
../accel/kvm/kvm-all.c:1837
7 0x560640c98f8e in virtio_pci_one_vector_unmask
(proxy=0x560643c65f00, queue_no=4294967295, vector=0, msg=...,
n=0x560643c6e4c8)
at ../hw/virtio/virtio-pci.c:1005
8 0x560640c99201 in virtio_pci_vector_unmask (dev=0x560643c65f00,
vector=0, msg=...)
at ../hw/virtio/virtio-pci.c:1070
9 0x560640bc402e in msix_fire_vector_notifier (dev=0x560643c65f00,
vector=0, is_masked=false)
at ../hw/pci/msix.c:120
10 0x560640bc40f1 in msix_handle_mask_update (dev=0x560643c65f00, vector=0,
was_masked=true)
at ../hw/pci/msix.c:140
11 0x560640bc4503 in msix_table_mmio_write (opaque=0x560643c65f00, addr=12,
val=0, size=4)
at ../hw/pci/msix.c:231
12 0x560640f26d83 in memory_region_write_accessor
(mr=0x560643c66540, addr=12, value=0x7fc86b7bc628, size=4, shift=0,
mask=4294967295, attrs=...)
at ../system/memory.c:497
13 0x560640f270a6 in access_with_adjusted_size
(addr=12, value=0x7fc86b7bc628, size=4, access_size_min=1,
access_size_max=4, access_fn=0x560640f26c8d ,
mr=0x560643c66540, attrs=...) at ../system/memory.c:573
14 0x560640f2a2b5 in memory_region_dispatch_write (mr=0x560643c66540,
addr=12, data=0, op=MO_32, attrs=...)
at ../system/memory.c:1521
15 0x560640f37bac in flatview_write_continue
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., ptr=0x7fc871e9c028, len=4,
addr1=12, l=4, mr=0x560643c66540)
at ../system/physmem.c:2714
16 0x560640f37d0f in flatview_write
(fv=0x7fc65805e0b0, addr=4273803276, attrs=..., buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2756
17 0x560640f380bf in address_space_write
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4)
at ../system/physmem.c:2863
18 0x560640f3812c in address_space_rw
(as=0x560642161ae0 , addr=4273803276, attrs=...,
buf=0x7fc871e9c028, len=4, is_write=true) at ../system/physmem.c:2873
--Type for more, q to quit, c to continue without paging--
19 0x560640f8aa55 in kvm_cpu_exec (cpu=0x560642f205e0) at
../accel/kvm/kvm-all.c:2915
20 0x560640f8d731 in kvm_vcpu_thread_fn (arg=0x560642f205e0) at
../accel/kvm/kvm-accel-ops.c:51
21 0x5606411949f4 in qemu_thread_start (args=0x560642f292b0) at
../util/qemu-thread-posix.c:541
22 0x7fc87148cdcd in start_thread (arg=) at
pthread_create.c:442
23 0x7fc871512630 in clone3 () at
../sysdeps/unix/sysv/linux/x86_64/clone3.S:81
(gdb)
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 35 +++
1 file changed, 35 insertions(+)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 1a7039fb0c..344f4fb844 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -880,6 +880,7 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
*proxy, int queue_no)
int ret;
EventNotifier *n;
PCIDevice *dev = >pci_dev;
+VirtIOIRQFD *irqfd;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
@@ -890,10 +891,19 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
*proxy, int queue_no)
if (vector >= msix_nr_vectors_allocated(dev)) {
return 0;
}
+/*
+ * if the irqfd still in use, means the irqfd was not
+ * release before and don't need to set up
+ */
+irqfd = >vector_irqfd[vector];
+if (irqfd->users != 0) {
+return 0;
+}
ret = kvm_virtio_pci_vq_vector_use(proxy, vector
Re: [PATCH 1/1] virtio-pci: Fix the crash when the vector changes back from VIRTIO_NO_VECTOR
On Mon, Apr 8, 2024 at 12:59 PM Jason Wang wrote:
>
> On Sun, Apr 7, 2024 at 3:00 PM Cindy Lu wrote:
> >
> > On Sun, Apr 7, 2024 at 12:20 PM Jason Wang wrote:
> > >
> > > On Tue, Apr 2, 2024 at 11:02 PM Cindy Lu wrote:
> > > >
> > > > When the guest calls virtio_stop and then virtio_reset,
> > >
> > > Guests could not call those functions directly, it is triggered by for
> > > example writing to some of the registers like reset or others.
> > >
> > sure , Will fix this
> > > > the vector will change
> > > > to VIRTIO_NO_VECTOR and the IRQFD for this vector will be released.
> > > > After that
> > > > If you want to change the vector back,
> > >
> > > What do you mean by "change the vector back"? Something like
> > >
> > > assign VIRTIO_MSI_NO_VECTOR to vector 0
> > > assign X to vector 0
> > >
> > yes, the process is something like
> >
> > set config_vector = VIRTIO_MSI_NO_VECTOR
> > ...
> > set config_vector = 0
> > > And I guess what you meant is to configure the vector after DRIVER_OK.
> >
> > >
> > >
> > > > it will cause a crash.
> > > >
> > > > To fix this, we need to call the function
> > > > "kvm_virtio_pci_vector_use_one()"
> > > > when the vector changes back from VIRTIO_NO_VECTOR
> > > >
> > > > Signed-off-by: Cindy Lu
> > > > ---
> > > > hw/virtio/virtio-pci.c | 31 ---
> > > > 1 file changed, 28 insertions(+), 3 deletions(-)
> > > >
> > > > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > > > index e433879542..45f3ab38c3 100644
> > > > --- a/hw/virtio/virtio-pci.c
> > > > +++ b/hw/virtio/virtio-pci.c
> > > > @@ -874,12 +874,14 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy
> > > > *proxy, int queue_no,
> > > > return 0;
> > > > }
> > > >
> > > > -static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int
> > > > queue_no)
> > > > +static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int
> > > > queue_no,
> > > > + bool recovery)
> > > > {
> > > > unsigned int vector;
> > > > int ret;
> > > > EventNotifier *n;
> > > > PCIDevice *dev = >pci_dev;
> > > > +VirtIOIRQFD *irqfd;
> > > > VirtIODevice *vdev = virtio_bus_get_device(>bus);
> > > > VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
> > > >
> > > > @@ -890,10 +892,21 @@ static int
> > > > kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
> > > > if (vector >= msix_nr_vectors_allocated(dev)) {
> > > > return 0;
> > > > }
> > > > +/*
> > > > + * if this is recovery and irqfd still in use, means the irqfd was
> > > > not
> > > > + * release before and don't need to set up again
> > > > + */
> > > > +if (recovery) {
> > > > +irqfd = >vector_irqfd[vector];
> > > > +if (irqfd->users != 0) {
> > > > +return 0;
> > > > +}
> > > > +}
> > > > ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
> > > > if (ret < 0) {
> > > > goto undo;
> > > > }
> > > > +
> > > > /*
> > > > * If guest supports masking, set up irqfd now.
> > > > * Otherwise, delay until unmasked in the frontend.
> > > > @@ -932,14 +945,14 @@ static int
> > > > kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy *proxy, int nvqs)
> > > > if (!virtio_queue_get_num(vdev, queue_no)) {
> > > > return -1;
> > > > }
> > > > -ret = kvm_virtio_pci_vector_use_one(proxy, queue_no);
> > > > +ret = kvm_virtio_pci_vector_use_one(proxy, queue_no, false);
> > > > }
> > > > return ret;
> > > > }
> > > >
> > > > static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
> > > > {
> > > > -return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
> > > > +re
Re: [PATCH 1/1] virtio-pci: Fix the crash when the vector changes back from VIRTIO_NO_VECTOR
On Sun, Apr 7, 2024 at 12:20 PM Jason Wang wrote:
>
> On Tue, Apr 2, 2024 at 11:02 PM Cindy Lu wrote:
> >
> > When the guest calls virtio_stop and then virtio_reset,
>
> Guests could not call those functions directly, it is triggered by for
> example writing to some of the registers like reset or others.
>
sure , Will fix this
> > the vector will change
> > to VIRTIO_NO_VECTOR and the IRQFD for this vector will be released. After
> > that
> > If you want to change the vector back,
>
> What do you mean by "change the vector back"? Something like
>
> assign VIRTIO_MSI_NO_VECTOR to vector 0
> assign X to vector 0
>
yes, the process is something like
set config_vector = VIRTIO_MSI_NO_VECTOR
...
set config_vector = 0
> And I guess what you meant is to configure the vector after DRIVER_OK.
>
>
> > it will cause a crash.
> >
> > To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()"
> > when the vector changes back from VIRTIO_NO_VECTOR
> >
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/virtio-pci.c | 31 ---
> > 1 file changed, 28 insertions(+), 3 deletions(-)
> >
> > diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
> > index e433879542..45f3ab38c3 100644
> > --- a/hw/virtio/virtio-pci.c
> > +++ b/hw/virtio/virtio-pci.c
> > @@ -874,12 +874,14 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy
> > *proxy, int queue_no,
> > return 0;
> > }
> >
> > -static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int
> > queue_no)
> > +static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int
> > queue_no,
> > + bool recovery)
> > {
> > unsigned int vector;
> > int ret;
> > EventNotifier *n;
> > PCIDevice *dev = >pci_dev;
> > +VirtIOIRQFD *irqfd;
> > VirtIODevice *vdev = virtio_bus_get_device(>bus);
> > VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
> >
> > @@ -890,10 +892,21 @@ static int
> > kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
> > if (vector >= msix_nr_vectors_allocated(dev)) {
> > return 0;
> > }
> > +/*
> > + * if this is recovery and irqfd still in use, means the irqfd was not
> > + * release before and don't need to set up again
> > + */
> > +if (recovery) {
> > +irqfd = >vector_irqfd[vector];
> > +if (irqfd->users != 0) {
> > +return 0;
> > +}
> > +}
> > ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
> > if (ret < 0) {
> > goto undo;
> > }
> > +
> > /*
> > * If guest supports masking, set up irqfd now.
> > * Otherwise, delay until unmasked in the frontend.
> > @@ -932,14 +945,14 @@ static int
> > kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy *proxy, int nvqs)
> > if (!virtio_queue_get_num(vdev, queue_no)) {
> > return -1;
> > }
> > -ret = kvm_virtio_pci_vector_use_one(proxy, queue_no);
> > +ret = kvm_virtio_pci_vector_use_one(proxy, queue_no, false);
> > }
> > return ret;
> > }
> >
> > static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
> > {
> > -return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
> > +return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX,
> > false);
> > }
> >
> > static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
> > @@ -1570,7 +1583,13 @@ static void virtio_pci_common_write(void *opaque,
> > hwaddr addr,
> > } else {
> > val = VIRTIO_NO_VECTOR;
> > }
> > +vector = vdev->config_vector;
> > vdev->config_vector = val;
> > +/*check if the vector need to recovery*/
> > +if ((val != VIRTIO_NO_VECTOR) && (vector == VIRTIO_NO_VECTOR) &&
> > +(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
> > +kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX,
> > true);
> > +}
>
> This looks too tricky.
>
> Think hard of this. I think it's better to split this into two parts:
>
> 1) a series that disables config irqfd for vhost-net, this series
> needs to be backported to -stable which needs to be conservative. It
> looks more like your V1, but let's add a boolean
[PATCH 0/1] virtio-pci: Fix the crash when the vector changes back from VIRTIO_NO_VECTOR
There is a crash in the Non-standard guest image. The root cause of the issue is that an IRQFD was used After it release During the booting process of the Vyatta image, the behavior of the called function in qemu is as follows: 1. vhost_net_stop() was called. This will call the function virtio_pci_set_guest_notifiers() with assgin= false, and virtio_pci_set_guest_notifiers(??? will release the irqfd for vector 0 2. virtio_reset() was called -->set configure vector to VIRTIO_NO_VECTOR 3.vhost_net_start() was called (at this time the configure vector is still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers() with assgin= true, so the irqfd for vector 0 was not "init" during this process 4. The system continues to boot, and msix_fire_vector_notifier() was called unmask the vector 0 and then met the crash [msix_fire_vector_notifier] 112 called vector 0 is_masked 1 [msix_fire_vector_notifier] 112 called vector 0 is_masked 0 To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()" when the vector changes back from VIRTIO_NO_VECTOR Signed-off-by: Cindy Lu Cindy Lu (1): virtio-pci: Fix the crash when the vector changes back from VIRTIO_NO_VECTOR hw/virtio/virtio-pci.c | 31 --- 1 file changed, 28 insertions(+), 3 deletions(-) -- 2.43.0
[PATCH 1/1] virtio-pci: Fix the crash when the vector changes back from VIRTIO_NO_VECTOR
When the guest calls virtio_stop and then virtio_reset, the vector will change
to VIRTIO_NO_VECTOR and the IRQFD for this vector will be released. After that
If you want to change the vector back, it will cause a crash.
To fix this, we need to call the function "kvm_virtio_pci_vector_use_one()"
when the vector changes back from VIRTIO_NO_VECTOR
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 31 ---
1 file changed, 28 insertions(+), 3 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index e433879542..45f3ab38c3 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -874,12 +874,14 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy *proxy,
int queue_no,
return 0;
}
-static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
+static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no,
+ bool recovery)
{
unsigned int vector;
int ret;
EventNotifier *n;
PCIDevice *dev = >pci_dev;
+VirtIOIRQFD *irqfd;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
@@ -890,10 +892,21 @@ static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy
*proxy, int queue_no)
if (vector >= msix_nr_vectors_allocated(dev)) {
return 0;
}
+/*
+ * if this is recovery and irqfd still in use, means the irqfd was not
+ * release before and don't need to set up again
+ */
+if (recovery) {
+irqfd = >vector_irqfd[vector];
+if (irqfd->users != 0) {
+return 0;
+}
+}
ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
if (ret < 0) {
goto undo;
}
+
/*
* If guest supports masking, set up irqfd now.
* Otherwise, delay until unmasked in the frontend.
@@ -932,14 +945,14 @@ static int kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy
*proxy, int nvqs)
if (!virtio_queue_get_num(vdev, queue_no)) {
return -1;
}
-ret = kvm_virtio_pci_vector_use_one(proxy, queue_no);
+ret = kvm_virtio_pci_vector_use_one(proxy, queue_no, false);
}
return ret;
}
static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
{
-return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
+return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX, false);
}
static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
@@ -1570,7 +1583,13 @@ static void virtio_pci_common_write(void *opaque, hwaddr
addr,
} else {
val = VIRTIO_NO_VECTOR;
}
+vector = vdev->config_vector;
vdev->config_vector = val;
+/*check if the vector need to recovery*/
+if ((val != VIRTIO_NO_VECTOR) && (vector == VIRTIO_NO_VECTOR) &&
+(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
+kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX, true);
+}
break;
case VIRTIO_PCI_COMMON_STATUS:
if (!(val & VIRTIO_CONFIG_S_DRIVER_OK)) {
@@ -1611,6 +1630,12 @@ static void virtio_pci_common_write(void *opaque, hwaddr
addr,
val = VIRTIO_NO_VECTOR;
}
virtio_queue_set_vector(vdev, vdev->queue_sel, val);
+
+/*check if the vector need to recovery*/
+if ((val != VIRTIO_NO_VECTOR) && (vector == VIRTIO_NO_VECTOR) &&
+(vdev->status & VIRTIO_CONFIG_S_DRIVER_OK)) {
+kvm_virtio_pci_vector_use_one(proxy, vdev->queue_sel, true);
+}
break;
case VIRTIO_PCI_COMMON_Q_ENABLE:
if (val == 1) {
--
2.43.0
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Thu, Mar 28, 2024 at 12:12 PM Jason Wang wrote:
>
> On Wed, Mar 27, 2024 at 5:33 PM Cindy Lu wrote:
> >
> > On Wed, Mar 27, 2024 at 5:12 PM Jason Wang wrote:
> > >
> > > On Wed, Mar 27, 2024 at 4:28 PM Cindy Lu wrote:
> > > >
> > > > On Wed, Mar 27, 2024 at 3:54 PM Jason Wang wrote:
> > > > >
> > > > > On Wed, Mar 27, 2024 at 2:03 PM Cindy Lu wrote:
> > > > > >
> > > > > > On Wed, Mar 27, 2024 at 11:05 AM Jason Wang
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi Cindy:
> > > > > > >
> > > > > > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu wrote:
> > > > > > > >
> > > > > > > > we need a crash in Non-standard image, here is the jira for
> > > > > > > > this https://issues.redhat.com/browse/RHEL-28522
> > > > > > > > The root cause of the issue is that an IRQFD was used without
> > > > > > > > initialization..
> > > > > > > >
> > > > > > > > During the booting process of the Vyatta image, the behavior of
> > > > > > > > the called function in qemu is as follows:
> > > > > > > >
> > > > > > > > 1. vhost_net_stop() was called, this will call the function
> > > > > > > > virtio_pci_set_guest_notifiers() with assgin= false, and
> > > > > > > > virtio_pci_set_guest_notifiers(） will release the irqfd for
> > > > > > > > vector 0
> > > > > > >
> > > > > > > Before vhost_net_stop(), do we know which vector is used by which
> > > > > > > queue?
> > > > > > >
> > > > > > before this stop, vdev->config_verctor is get from
> > > > > > virtio_pci_common_read/virtio_pci_common_write
> > > > > > it was set to vector 0
> > > > >
> > > > > I basically meant if vector 0 is shared with some virtqueues here.
> > > > >
> > > > Really sorry for this, vq's vector is 1,2, and will not share with the
> > > > configure vector
> > > > > > > >
> > > > > > > > 2. virtio_reset() was called -->set configure vector to
> > > > > > > > VIRTIO_NO_VECTORt
> > > > > > > >
> > > > > > > > 3.vhost_net_start() was called (at this time the configure
> > > > > > > > vector is
> > > > > > > > still VIRTIO_NO_VECTOR) and call
> > > > > > > > virtio_pci_set_guest_notifiers() with
> > > > > > > > assgin= true, so the irqfd for vector 0 was not "init" during
> > > > > > > > this process
> > > > > > >
> > > > > > > How does the configure vector differ from the virtqueue vector
> > > > > > > here?
> > > > > > >
> > > > > > All the vectors are VIRTIO_NO_VECTOR (including vq). any
> > > > > > msix_fire_vector_notifier()
> > > > > > been called will cause the crash at this time.
> > > > >
> > > > > Won't virtio_pci_set_guest_notifiers() will try to allocate irqfd when
> > > > > the assignment is true?
> > > > >
> > > > It will allocate, but the vector is VIRTIO_NO_VECTOR (0x)
> > > >
> > > > then it will called kvm_virtio_pci_vector_use_one()
> > > >
> > > > in this function, there is a check for
> > > >
> > > > if (vector >= msix_nr_vectors_allocated(dev))
> > > >
> > > > { return 0; }
> > > >
> > > > So it will return.
> > >
> > > How about let's just fix this?
> > >
> > > Btw, it's better to explain in detail like the above in the next version.
> > >
> > > Thanks
> > >
> > The problem is I think the behavior here is correct, The vector here is
> > VIRTIO_NO_VECTOR and we should return,
>
> So if I understand correctly, the configure vector is configured after
> DRIVER_OK?
>
sorry I didn't get your point, Do you mean set_guest_notifiers()?,
this was called during the system boot
but for the value of vdev->config_vector/vq vector, this is changed
by virtio_pci_common_read/
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Thu, Mar 28, 2024 at 12:14 PM Jason Wang wrote:
>
> On Wed, Mar 27, 2024 at 5:44 PM Cindy Lu wrote:
> >
> > On Wed, Mar 27, 2024 at 5:13 PM Jason Wang wrote:
> > >
> > > On Wed, Mar 27, 2024 at 5:12 PM Jason Wang wrote:
> > > >
> > > > On Wed, Mar 27, 2024 at 4:28 PM Cindy Lu wrote:
> > > > >
> > > > > On Wed, Mar 27, 2024 at 3:54 PM Jason Wang
> > > > > wrote:
> > > > > >
> > > > > > On Wed, Mar 27, 2024 at 2:03 PM Cindy Lu wrote:
> > > > > > >
> > > > > > > On Wed, Mar 27, 2024 at 11:05 AM Jason Wang
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi Cindy:
> > > > > > > >
> > > > > > > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > we need a crash in Non-standard image, here is the jira for
> > > > > > > > > this https://issues.redhat.com/browse/RHEL-28522
> > > > > > > > > The root cause of the issue is that an IRQFD was used without
> > > > > > > > > initialization..
> > > > > > > > >
> > > > > > > > > During the booting process of the Vyatta image, the behavior
> > > > > > > > > of the called function in qemu is as follows:
> > > > > > > > >
> > > > > > > > > 1. vhost_net_stop() was called, this will call the function
> > > > > > > > > virtio_pci_set_guest_notifiers() with assgin= false, and
> > > > > > > > > virtio_pci_set_guest_notifiers(） will release the irqfd for
> > > > > > > > > vector 0
> > > > > > > >
> > > > > > > > Before vhost_net_stop(), do we know which vector is used by
> > > > > > > > which queue?
> > > > > > > >
> > > > > > > before this stop, vdev->config_verctor is get from
> > > > > > > virtio_pci_common_read/virtio_pci_common_write
> > > > > > > it was set to vector 0
> > > > > >
> > > > > > I basically meant if vector 0 is shared with some virtqueues here.
> > > > > >
> > > > > Really sorry for this, vq's vector is 1,2, and will not share with the
> > > > > configure vector
> > > > > > > > >
> > > > > > > > > 2. virtio_reset() was called -->set configure vector to
> > > > > > > > > VIRTIO_NO_VECTORt
> > > > > > > > >
> > > > > > > > > 3.vhost_net_start() was called (at this time the configure
> > > > > > > > > vector is
> > > > > > > > > still VIRTIO_NO_VECTOR) and call
> > > > > > > > > virtio_pci_set_guest_notifiers() with
> > > > > > > > > assgin= true, so the irqfd for vector 0 was not "init" during
> > > > > > > > > this process
> > > > > > > >
> > > > > > > > How does the configure vector differ from the virtqueue vector
> > > > > > > > here?
> > > > > > > >
> > > > > > > All the vectors are VIRTIO_NO_VECTOR (including vq). any
> > > > > > > msix_fire_vector_notifier()
> > > > > > > been called will cause the crash at this time.
> > > > > >
> > > > > > Won't virtio_pci_set_guest_notifiers() will try to allocate irqfd
> > > > > > when
> > > > > > the assignment is true?
> > > > > >
> > > > > It will allocate, but the vector is VIRTIO_NO_VECTOR (0x)
> > > > >
> > > > > then it will called kvm_virtio_pci_vector_use_one()
> > > > >
> > > > > in this function, there is a check for
> > > > >
> > > > > if (vector >= msix_nr_vectors_allocated(dev))
> > > > >
> > > > > { return 0; }
> > > > >
> > > > > So it will return.
> > > >
> > > > How about let's just fix this?
> > >
> > > Btw, another question
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Wed, Mar 27, 2024 at 5:13 PM Jason Wang wrote:
>
> On Wed, Mar 27, 2024 at 5:12 PM Jason Wang wrote:
> >
> > On Wed, Mar 27, 2024 at 4:28 PM Cindy Lu wrote:
> > >
> > > On Wed, Mar 27, 2024 at 3:54 PM Jason Wang wrote:
> > > >
> > > > On Wed, Mar 27, 2024 at 2:03 PM Cindy Lu wrote:
> > > > >
> > > > > On Wed, Mar 27, 2024 at 11:05 AM Jason Wang
> > > > > wrote:
> > > > > >
> > > > > > Hi Cindy:
> > > > > >
> > > > > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu wrote:
> > > > > > >
> > > > > > > we need a crash in Non-standard image, here is the jira for this
> > > > > > > https://issues.redhat.com/browse/RHEL-28522
> > > > > > > The root cause of the issue is that an IRQFD was used without
> > > > > > > initialization..
> > > > > > >
> > > > > > > During the booting process of the Vyatta image, the behavior of
> > > > > > > the called function in qemu is as follows:
> > > > > > >
> > > > > > > 1. vhost_net_stop() was called, this will call the function
> > > > > > > virtio_pci_set_guest_notifiers() with assgin= false, and
> > > > > > > virtio_pci_set_guest_notifiers(） will release the irqfd for
> > > > > > > vector 0
> > > > > >
> > > > > > Before vhost_net_stop(), do we know which vector is used by which
> > > > > > queue?
> > > > > >
> > > > > before this stop, vdev->config_verctor is get from
> > > > > virtio_pci_common_read/virtio_pci_common_write
> > > > > it was set to vector 0
> > > >
> > > > I basically meant if vector 0 is shared with some virtqueues here.
> > > >
> > > Really sorry for this, vq's vector is 1,2, and will not share with the
> > > configure vector
> > > > > > >
> > > > > > > 2. virtio_reset() was called -->set configure vector to
> > > > > > > VIRTIO_NO_VECTORt
> > > > > > >
> > > > > > > 3.vhost_net_start() was called (at this time the configure vector
> > > > > > > is
> > > > > > > still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers()
> > > > > > > with
> > > > > > > assgin= true, so the irqfd for vector 0 was not "init" during
> > > > > > > this process
> > > > > >
> > > > > > How does the configure vector differ from the virtqueue vector here?
> > > > > >
> > > > > All the vectors are VIRTIO_NO_VECTOR (including vq). any
> > > > > msix_fire_vector_notifier()
> > > > > been called will cause the crash at this time.
> > > >
> > > > Won't virtio_pci_set_guest_notifiers() will try to allocate irqfd when
> > > > the assignment is true?
> > > >
> > > It will allocate, but the vector is VIRTIO_NO_VECTOR (0x)
> > >
> > > then it will called kvm_virtio_pci_vector_use_one()
> > >
> > > in this function, there is a check for
> > >
> > > if (vector >= msix_nr_vectors_allocated(dev))
> > >
> > > { return 0; }
> > >
> > > So it will return.
> >
> > How about let's just fix this?
>
> Btw, another question, how does vDPA work here?
>
> Thanks
>
the rhel/fedroa guest image will not call vrtio_stop and virtio_reset
during the boot
So vector will not change to VIRTIO_NO_VECTOR. So the vdpa's
configure interrupt
Should work ok and there is no crash
Thanks
cindy
> >
> > Btw, it's better to explain in detail like the above in the next version.
> >
> > Thanks
> >
> > >
> > > > > So I think this should
> > > > > be a bug in this guest image
> > > >
> > > > The point is Qemu should not crash even if the guest driver is buggy.
> > > >
> > > > It would be nice if we can have a qtest for this on top.
> > > >
> > > > Thanks
> > > >
> > > sure, got it, I have done the Qtest, and it passed
> > > here is the result
> > >
> > > Ok: 794
> > > Expected Fail: 0
> > > Fail: 0
> > > Unexpected Pass:0
> >
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Wed, Mar 27, 2024 at 5:12 PM Jason Wang wrote:
>
> On Wed, Mar 27, 2024 at 4:28 PM Cindy Lu wrote:
> >
> > On Wed, Mar 27, 2024 at 3:54 PM Jason Wang wrote:
> > >
> > > On Wed, Mar 27, 2024 at 2:03 PM Cindy Lu wrote:
> > > >
> > > > On Wed, Mar 27, 2024 at 11:05 AM Jason Wang wrote:
> > > > >
> > > > > Hi Cindy:
> > > > >
> > > > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu wrote:
> > > > > >
> > > > > > we need a crash in Non-standard image, here is the jira for this
> > > > > > https://issues.redhat.com/browse/RHEL-28522
> > > > > > The root cause of the issue is that an IRQFD was used without
> > > > > > initialization..
> > > > > >
> > > > > > During the booting process of the Vyatta image, the behavior of the
> > > > > > called function in qemu is as follows:
> > > > > >
> > > > > > 1. vhost_net_stop() was called, this will call the function
> > > > > > virtio_pci_set_guest_notifiers() with assgin= false, and
> > > > > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
> > > > >
> > > > > Before vhost_net_stop(), do we know which vector is used by which
> > > > > queue?
> > > > >
> > > > before this stop, vdev->config_verctor is get from
> > > > virtio_pci_common_read/virtio_pci_common_write
> > > > it was set to vector 0
> > >
> > > I basically meant if vector 0 is shared with some virtqueues here.
> > >
> > Really sorry for this, vq's vector is 1,2, and will not share with the
> > configure vector
> > > > > >
> > > > > > 2. virtio_reset() was called -->set configure vector to
> > > > > > VIRTIO_NO_VECTORt
> > > > > >
> > > > > > 3.vhost_net_start() was called (at this time the configure vector is
> > > > > > still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers()
> > > > > > with
> > > > > > assgin= true, so the irqfd for vector 0 was not "init" during this
> > > > > > process
> > > > >
> > > > > How does the configure vector differ from the virtqueue vector here?
> > > > >
> > > > All the vectors are VIRTIO_NO_VECTOR (including vq). any
> > > > msix_fire_vector_notifier()
> > > > been called will cause the crash at this time.
> > >
> > > Won't virtio_pci_set_guest_notifiers() will try to allocate irqfd when
> > > the assignment is true?
> > >
> > It will allocate, but the vector is VIRTIO_NO_VECTOR (0x)
> >
> > then it will called kvm_virtio_pci_vector_use_one()
> >
> > in this function, there is a check for
> >
> > if (vector >= msix_nr_vectors_allocated(dev))
> >
> > { return 0; }
> >
> > So it will return.
>
> How about let's just fix this?
>
> Btw, it's better to explain in detail like the above in the next version.
>
> Thanks
>
The problem is I think the behavior here is correct, The vector here is
VIRTIO_NO_VECTOR and we should return,
the fix could work maybe is we try get to know if this was changed
from another value
and use that one? this seems strange.
Thanks
cindy
> >
> > > > So I think this should
> > > > be a bug in this guest image
> > >
> > > The point is Qemu should not crash even if the guest driver is buggy.
> > >
> > > It would be nice if we can have a qtest for this on top.
> > >
> > > Thanks
> > >
> > sure, got it, I have done the Qtest, and it passed
> > here is the result
> >
> > Ok: 794
> > Expected Fail: 0
> > Fail: 0
> > Unexpected Pass:0
> > Skipped:32
> > Timeout:0
> >
> > > > > >
> > > > > > 4. The system continues to boot and msix_fire_vector_notifier() was
> > > > > > called unmask the vector 0 and then met the crash
> > > > > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 1
> > > > > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 0
> > > > > >
> > > > > > The reason for not reproducing in RHEL/fedora guest image is because
> > > > > > REHL/Fedora doesn't ha
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Wed, Mar 27, 2024 at 3:54 PM Jason Wang wrote:
>
> On Wed, Mar 27, 2024 at 2:03 PM Cindy Lu wrote:
> >
> > On Wed, Mar 27, 2024 at 11:05 AM Jason Wang wrote:
> > >
> > > Hi Cindy:
> > >
> > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu wrote:
> > > >
> > > > we need a crash in Non-standard image, here is the jira for this
> > > > https://issues.redhat.com/browse/RHEL-28522
> > > > The root cause of the issue is that an IRQFD was used without
> > > > initialization..
> > > >
> > > > During the booting process of the Vyatta image, the behavior of the
> > > > called function in qemu is as follows:
> > > >
> > > > 1. vhost_net_stop() was called, this will call the function
> > > > virtio_pci_set_guest_notifiers() with assgin= false, and
> > > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0
> > >
> > > Before vhost_net_stop(), do we know which vector is used by which queue?
> > >
> > before this stop, vdev->config_verctor is get from
> > virtio_pci_common_read/virtio_pci_common_write
> > it was set to vector 0
>
> I basically meant if vector 0 is shared with some virtqueues here.
>
Really sorry for this, vq's vector is 1,2, and will not share with the
configure vector
> > > >
> > > > 2. virtio_reset() was called -->set configure vector to
> > > > VIRTIO_NO_VECTORt
> > > >
> > > > 3.vhost_net_start() was called (at this time the configure vector is
> > > > still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers() with
> > > > assgin= true, so the irqfd for vector 0 was not "init" during this
> > > > process
> > >
> > > How does the configure vector differ from the virtqueue vector here?
> > >
> > All the vectors are VIRTIO_NO_VECTOR (including vq). any
> > msix_fire_vector_notifier()
> > been called will cause the crash at this time.
>
> Won't virtio_pci_set_guest_notifiers() will try to allocate irqfd when
> the assignment is true?
>
It will allocate, but the vector is VIRTIO_NO_VECTOR (0x)
then it will called kvm_virtio_pci_vector_use_one()
in this function, there is a check for
if (vector >= msix_nr_vectors_allocated(dev))
{ return 0; }
So it will return.
> > So I think this should
> > be a bug in this guest image
>
> The point is Qemu should not crash even if the guest driver is buggy.
>
> It would be nice if we can have a qtest for this on top.
>
> Thanks
>
sure, got it, I have done the Qtest, and it passed
here is the result
Ok: 794
Expected Fail: 0
Fail: 0
Unexpected Pass:0
Skipped:32
Timeout:0
> > > >
> > > > 4. The system continues to boot and msix_fire_vector_notifier() was
> > > > called unmask the vector 0 and then met the crash
> > > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 1
> > > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 0
> > > >
> > > > The reason for not reproducing in RHEL/fedora guest image is because
> > > > REHL/Fedora doesn't have the behavior of calling vhost_net_stop and
> > > > then virtio_reset, and also won't call msix_fire_vector_notifier for
> > > > vector 0 during system boot.
> > > >
> > > > The reason for not reproducing before configure interrupt support is
> > > > because
> > > > vector 0 is for configure interrupt, before the support for configure
> > > > interrupts, the notifier process will not handle vector 0.
> > > >
> > > > For the device Vyatta using, it doesn't support configure interrupts at
> > > > all, So we plan to disable the configure interrupts in unsupported
> > > > device
> > >
> > > Btw, let's tweak the changelog, it's a little bit hard to understand.
> > >
> > sure will do
> > thanks
> > Cindy
> > > Thanks
> > >
> > > >
> > > > Signed-off-by: Cindy Lu
> > > >
> > > > Cindy Lu (2):
> > > > virtio-net: disable the configure interrupt for not support device
> > > > virtio-pci: check if the configure interrupt enable
> > > >
> > > > hw/net/virtio-net.c| 5 -
> > > > hw/virtio/virtio-pci.c | 41 +-
> > > > hw/virtio/virtio.c | 1 +
> > > > include/hw/virtio/virtio.h | 1 +
> > > > 4 files changed, 29 insertions(+), 19 deletions(-)
> > > >
> > > > --
> > > > 2.43.0
> > > >
> > >
> >
>
Re: [RFC 0/2] disable the configuration interrupt for the unsupported device
On Wed, Mar 27, 2024 at 11:05 AM Jason Wang wrote: > > Hi Cindy: > > On Wed, Mar 27, 2024 at 9:29 AM Cindy Lu wrote: > > > > we need a crash in Non-standard image, here is the jira for this > > https://issues.redhat.com/browse/RHEL-28522 > > The root cause of the issue is that an IRQFD was used without > > initialization.. > > > > During the booting process of the Vyatta image, the behavior of the called > > function in qemu is as follows: > > > > 1. vhost_net_stop() was called, this will call the function > > virtio_pci_set_guest_notifiers() with assgin= false, and > > virtio_pci_set_guest_notifiers(） will release the irqfd for vector 0 > > Before vhost_net_stop(), do we know which vector is used by which queue? > before this stop, vdev->config_verctor is get from virtio_pci_common_read/virtio_pci_common_write it was set to vector 0 > > > > 2. virtio_reset() was called -->set configure vector to VIRTIO_NO_VECTORt > > > > 3.vhost_net_start() was called (at this time the configure vector is > > still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers() with > > assgin= true, so the irqfd for vector 0 was not "init" during this process > > How does the configure vector differ from the virtqueue vector here? > All the vectors are VIRTIO_NO_VECTOR (including vq). any msix_fire_vector_notifier() been called will cause the crash at this time. So I think this should be a bug in this guest image > > > > 4. The system continues to boot and msix_fire_vector_notifier() was > > called unmask the vector 0 and then met the crash > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 1 > > [msix_fire_vector_notifier] 112 called vector 0 is_masked 0 > > > > The reason for not reproducing in RHEL/fedora guest image is because > > REHL/Fedora doesn't have the behavior of calling vhost_net_stop and then > > virtio_reset, and also won't call msix_fire_vector_notifier for vector 0 > > during system boot. > > > > The reason for not reproducing before configure interrupt support is because > > vector 0 is for configure interrupt, before the support for configure > > interrupts, the notifier process will not handle vector 0. > > > > For the device Vyatta using, it doesn't support configure interrupts at > > all, So we plan to disable the configure interrupts in unsupported device > > Btw, let's tweak the changelog, it's a little bit hard to understand. > sure will do thanks Cindy > Thanks > > > > > Signed-off-by: Cindy Lu > > > > Cindy Lu (2): > > virtio-net: disable the configure interrupt for not support device > > virtio-pci: check if the configure interrupt enable > > > > hw/net/virtio-net.c| 5 - > > hw/virtio/virtio-pci.c | 41 +- > > hw/virtio/virtio.c | 1 + > > include/hw/virtio/virtio.h | 1 + > > 4 files changed, 29 insertions(+), 19 deletions(-) > > > > -- > > 2.43.0 > > >
[RFC 1/2] virtio-net: disable the configure interrupt for not support device
Only the vdpa device support configure interrupt, we need to disable the
configure interrupt process in all other device.
In order to achieve this, I added a check in the virtio_net_device_realize().
When the peer's type is vdpa, the value of config_irq_enabled in the structure
VirtIODevice will set to true.
Signed-off-by: Cindy Lu
---
hw/net/virtio-net.c| 5 -
hw/virtio/virtio.c | 1 +
include/hw/virtio/virtio.h | 1 +
3 files changed, 6 insertions(+), 1 deletion(-)
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index 80c56f0cfc..3b487864a8 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3749,12 +3749,15 @@ static void virtio_net_device_realize(DeviceState *dev,
Error **errp)
nc = qemu_get_queue(n->nic);
nc->rxfilter_notify_enabled = 1;
+vdev->config_irq_enabled = false;
- if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
+if (nc->peer && nc->peer->info->type == NET_CLIENT_DRIVER_VHOST_VDPA) {
struct virtio_net_config netcfg = {};
memcpy(, >nic_conf.macaddr, ETH_ALEN);
vhost_net_set_config(get_vhost_net(nc->peer),
(uint8_t *), 0, ETH_ALEN, VHOST_SET_CONFIG_TYPE_FRONTEND);
+
+vdev->config_irq_enabled = true;
}
QTAILQ_INIT(>rsc_chains);
n->qdev = dev;
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index 3a160f86ed..6b52a7190d 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -3255,6 +3255,7 @@ void virtio_init(VirtIODevice *vdev, uint16_t device_id,
size_t config_size)
virtio_vmstate_change, vdev);
vdev->device_endian = virtio_default_endian();
vdev->use_guest_notifier_mask = true;
+vdev->config_irq_enabled = false;
}
/*
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index c8f72850bc..a7763b71e0 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -164,6 +164,7 @@ struct VirtIODevice
*/
EventNotifier config_notifier;
bool device_iotlb_enabled;
+bool config_irq_enabled;
};
struct VirtioDeviceClass {
--
2.43.0
[RFC 2/2] virtio-pci: check if the configure interrupt enable
If config_irq_enabled is not true, it means that configure interrupt is
not supported. Therefore, the config vector will not be handled during
the interrupt process.
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 41 +++--
1 file changed, 23 insertions(+), 18 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index e433879542..36ad7da206 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -1065,7 +1065,7 @@ static int virtio_pci_vector_unmask(PCIDevice *dev,
unsigned vector,
vq = virtio_vector_next_queue(vq);
}
/* unmask config intr */
-if (vector == vdev->config_vector) {
+if ((vector == vdev->config_vector) && (true == vdev->config_irq_enabled))
{
n = virtio_config_get_guest_notifier(vdev);
ret = virtio_pci_one_vector_unmask(proxy, VIRTIO_CONFIG_IRQ_IDX,
vector,
msg, n);
@@ -,7 +,7 @@ static void virtio_pci_vector_mask(PCIDevice *dev,
unsigned vector)
vq = virtio_vector_next_queue(vq);
}
-if (vector == vdev->config_vector) {
+if ((vector == vdev->config_vector) && (true == vdev->config_irq_enabled))
{
n = virtio_config_get_guest_notifier(vdev);
virtio_pci_one_vector_mask(proxy, VIRTIO_CONFIG_IRQ_IDX, vector, n);
}
@@ -1147,21 +1147,24 @@ static void virtio_pci_vector_poll(PCIDevice *dev,
}
}
/* poll the config intr */
-ret = virtio_pci_get_notifier(proxy, VIRTIO_CONFIG_IRQ_IDX, ,
- );
-if (ret < 0) {
-return;
-}
-if (vector < vector_start || vector >= vector_end ||
-!msix_is_masked(dev, vector)) {
-return;
-}
-if (k->guest_notifier_pending) {
-if (k->guest_notifier_pending(vdev, VIRTIO_CONFIG_IRQ_IDX)) {
+if (true == vdev->config_irq_enabled) {
+ret = virtio_pci_get_notifier(proxy, VIRTIO_CONFIG_IRQ_IDX, ,
+ );
+if (ret < 0) {
+return;
+}
+
+if (vector < vector_start || vector >= vector_end ||
+!msix_is_masked(dev, vector)) {
+return;
+}
+if (k->guest_notifier_pending) {
+if (k->guest_notifier_pending(vdev, VIRTIO_CONFIG_IRQ_IDX)) {
+msix_set_pending(dev, vector);
+}
+} else if (event_notifier_test_and_clear(notifier)) {
msix_set_pending(dev, vector);
}
-} else if (event_notifier_test_and_clear(notifier)) {
-msix_set_pending(dev, vector);
}
}
@@ -1282,9 +1285,11 @@ static int virtio_pci_set_guest_notifiers(DeviceState
*d, int nvqs, bool assign)
if (r < 0) {
goto config_assign_error;
}
-r = kvm_virtio_pci_vector_config_use(proxy);
-if (r < 0) {
-goto config_error;
+if (true == vdev->config_irq_enabled) {
+r = kvm_virtio_pci_vector_config_use(proxy);
+if (r < 0) {
+goto config_error;
+}
}
}
--
2.43.0
[RFC 0/2] disable the configuration interrupt for the unsupported device
we need a crash in Non-standard image, here is the jira for this https://issues.redhat.com/browse/RHEL-28522 The root cause of the issue is that an IRQFD was used without initialization.. During the booting process of the Vyatta image, the behavior of the called function in qemu is as follows: 1. vhost_net_stop() was called, this will call the function virtio_pci_set_guest_notifiers() with assgin= false, and virtio_pci_set_guest_notifiers(??? will release the irqfd for vector 0 2. virtio_reset() was called -->set configure vector to VIRTIO_NO_VECTORt 3.vhost_net_start() was called (at this time the configure vector is still VIRTIO_NO_VECTOR) and call virtio_pci_set_guest_notifiers() with assgin= true, so the irqfd for vector 0 was not "init" during this process 4. The system continues to boot and msix_fire_vector_notifier() was called unmask the vector 0 and then met the crash [msix_fire_vector_notifier] 112 called vector 0 is_masked 1 [msix_fire_vector_notifier] 112 called vector 0 is_masked 0 The reason for not reproducing in RHEL/fedora guest image is because REHL/Fedora doesn't have the behavior of calling vhost_net_stop and then virtio_reset, and also won't call msix_fire_vector_notifier for vector 0 during system boot. The reason for not reproducing before configure interrupt support is because vector 0 is for configure interrupt, before the support for configure interrupts, the notifier process will not handle vector 0. For the device Vyatta using, it doesn't support configure interrupts at all, So we plan to disable the configure interrupts in unsupported device Signed-off-by: Cindy Lu Cindy Lu (2): virtio-net: disable the configure interrupt for not support device virtio-pci: check if the configure interrupt enable hw/net/virtio-net.c| 5 - hw/virtio/virtio-pci.c | 41 +- hw/virtio/virtio.c | 1 + include/hw/virtio/virtio.h | 1 + 4 files changed, 29 insertions(+), 19 deletions(-) -- 2.43.0
Re: Re: [PATCH] vdpa-dev: Fix initialisation order to restore VDUSE compatibility
Jason Wang 于2024年2月7日周三 11:17写道：
>
> On Tue, Feb 6, 2024 at 4:31 PM Stefano Garzarella wrote:
> >
> > On Tue, Feb 06, 2024 at 10:47:40AM +0800, Jason Wang wrote:
> > >On Mon, Feb 5, 2024 at 6:51 PM Stefano Garzarella
> > >wrote:
> > >>
> > >> On Fri, Feb 02, 2024 at 02:25:21PM +0100, Kevin Wolf wrote:
> > >> >VDUSE requires that virtqueues are first enabled before the DRIVER_OK
> > >> >status flag is set; with the current API of the kernel module, it is
> > >> >impossible to enable the opposite order in our block export code because
> > >> >userspace is not notified when a virtqueue is enabled.
> > >
> > >Did this mean virtio-blk will enable a virtqueue after DRIVER_OK?
> >
> > It's not specific to virtio-blk, but to the generic vdpa device we have
> > in QEMU (i.e. vhost-vdpa-device). Yep, after commit
> > 6c4825476a4351530bcac17abab72295b75ffe98, virtqueues are enabled after
> > DRIVER_OK.
>
> Right.
>
> >
> > >Sepc is not clear about this and that's why we introduce
> > >VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK.
> >
> > Ah, I didn't know about this new feature. So after commit
> > 6c4825476a4351530bcac17abab72295b75ffe98 the vhost-vdpa-device is not
> > complying with the specification, right?
>
> Kind of, but as stated, it's just because spec is unclear about the
> behaviour. There's a chance that spec will explicitly support it in
> the future.
>
> >
> > >
> > >>
> > >> Yeah, IMHO the VDUSE protocol is missing a VDUSE_SET_VQ_READY message,
> > >
> > >I think you meant when VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK is
> > >negotiated.
> >
> > At this point yes. But if VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK is not
> > negotiated, should we return an error in vhost-vdpa kernel module if
> > VHOST_VDPA_SET_VRING_ENABLE is called when DRIVER_OK is already set?
>
> I'm not sure if this can break some setups or not. It might be better
> to leave it as is?
>
> Without VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK, we don't know if
> parent support vq_ready after driver_ok.
> With VHOST_BACKEND_F_ENABLE_AFTER_DRIVER_OK, we know parent support
> vq_ready after driver_ok.
>
> >
> > >If this is truth, it seems a little more complicated, for
> > >example the get_backend_features needs to be forward to the userspace?
> >
> > I'm not understanding, don't we already have VHOST_GET_BACKEND_FEATURES
> > for this? Or do you mean userspace on the VDUSE side?
>
> Yes, since in this case the parent is in the userspace, there's no way
> for VDUSE to know if user space supports vq_ready after driver_ok or
> not.
>
> As you may have noticed, we don't have a message for vq_ready which
> implies that vq_ready after driver_ok can't be supported.
>
> >
> > >This seems suboptimal to implement this in the spec first and then we
> > >can leverage the features. Or we can have another parameter for the
> > >ioctl that creates the vduse device.
> >
> > I got a little lost, though in vhost-user, the device can always expect
> > a vring_enable/disable, so I thought it was not complicated in VDUSE.
>
> Yes, the problem is assuming we have a message for vq_ready, there
> could be a "legacy" userspace that doesn't support that. So in that
> case, VDUSE needs to know if the userspace parent can support that or
> not.
>
> >
> > >
> > >> I'll start another thread about that, but in the meantime I agree that
> > >> we should fix QEMU since we need to work properly with old kernels as
> > >> well.
> > >>
> > >> >
> > >> >This requirement also mathces the normal initialisation order as done by
> > >> >the generic vhost code in QEMU. However, commit 6c482547 accidentally
> > >> >changed the order for vdpa-dev and broke access to VDUSE devices with
> > >> >this.
> > >> >
> > >> >This changes vdpa-dev to use the normal order again and use the standard
> > >> >vhost callback .vhost_set_vring_enable for this. VDUSE devices can be
> > >> >used with vdpa-dev again after this fix.
> > >>
> > >> I like this approach and the patch LGTM, but I'm a bit worried about
> > >> this function in hw/net/vhost_net.c:
> > >>
> > >> int vhost_set_vring_enable(NetClientState *nc, int enable)
> > >> {
> > >> VHostNetState *net = get_vhost_net(nc);
> > >> const VhostOps *vhost_ops = net->dev.vhost_ops;
> > >>
> > >> nc->vring_enable = enable;
> > >>
> > >> if (vhost_ops && vhost_ops->vhost_set_vring_enable) {
> > >> return vhost_ops->vhost_set_vring_enable(>dev, enable);
> > >> }
> > >>
> > >> return 0;
> > >> }
> > >>
> > >> @Eugenio, @Jason, should we change some things there if vhost-vdpa
> > >> implements the vhost_set_vring_enable callback?
> > >
> > >Eugenio may know more, I remember we need to enable cvq first for
> > >shadow virtqueue to restore some states.
> > >
> > >>
> > >> Do you remember why we didn't implement it from the beginning?
> > >
> > >It seems the vrings parameter is introduced after vhost-vdpa is
> > >implemented.
> >
> > Sorry, I mean why we didn't implement the
Re: [RFC 0/7] vhost-vdpa: add support for iommufd
Hi Michael， Really sorry for the delay, I was sick-leave for almost 2 months, which caused the delay in the development of this feature. I will continue working on this feature soon. Thanks Cindy On Wed, Sep 13, 2023 at 9:31 PM Michael S. Tsirkin wrote: > > On Wed, May 03, 2023 at 05:13:30PM +0800, Cindy Lu wrote: > > Hi All > > There is the RFC to support the IOMMUFD in vdpa device > > any comments are welcome > > Thanks > > Cindy > > Any plans to work on this or should I consider this abandoned? > > > > Cindy Lu (7): > > vhost: introduce new UAPI to support IOMMUFD > > qapi: support iommufd in vdpa > > virtio : add a ptr for vdpa_iommufd in VirtIODevice > > net/vhost-vdpa: Add the check for iommufd > > vhost-vdpa: Add the iommufd support in the map/unmap function > > vhost-vdpa: init iommufd function in vhost_vdpa start > > vhost-vdpa-iommufd: Add iommufd support for vdpa > > > > hw/virtio/meson.build | 2 +- > > hw/virtio/vhost-vdpa-iommufd.c | 240 + > > hw/virtio/vhost-vdpa.c | 74 +- > > include/hw/virtio/vhost-vdpa.h | 47 +++ > > include/hw/virtio/virtio.h | 5 + > > linux-headers/linux/vhost.h| 72 ++ > > net/vhost-vdpa.c | 31 +++-- > > qapi/net.json | 1 + > > 8 files changed, 451 insertions(+), 21 deletions(-) > > create mode 100644 hw/virtio/vhost-vdpa-iommufd.c > > > > -- > > 2.34.3 >
Re: [RFC 0/7] vhost-vdpa: add support for iommufd
On Mon, Jun 5, 2023 at 1:41 PM Michael S. Tsirkin wrote: > > On Fri, May 05, 2023 at 02:29:23PM +0800, Cindy Lu wrote: > > On Fri, May 5, 2023 at 11:29 AM Jason Wang wrote: > > > > > > Hi Cindy > > > > > > On Wed, May 3, 2023 at 5:13 PM Cindy Lu wrote: > > > > > > > > Hi All > > > > There is the RFC to support the IOMMUFD in vdpa device > > > > any comments are welcome > > > > Thanks > > > > Cindy > > > > > > Please post the kernel patch as well as a reference. > > > > > > Thanks > > > > > sure，will do > > Thanks > > cindy > > Is this effort going anywhere? It will soon be too late for > the next merge window. > Hi Michael I'm now working on some vduse issue, I will go back to work in IOMMUFD soon Thanks Cindy > > > > > > > > Cindy Lu (7): > > > > vhost: introduce new UAPI to support IOMMUFD > > > > qapi: support iommufd in vdpa > > > > virtio : add a ptr for vdpa_iommufd in VirtIODevice > > > > net/vhost-vdpa: Add the check for iommufd > > > > vhost-vdpa: Add the iommufd support in the map/unmap function > > > > vhost-vdpa: init iommufd function in vhost_vdpa start > > > > vhost-vdpa-iommufd: Add iommufd support for vdpa > > > > > > > > hw/virtio/meson.build | 2 +- > > > > hw/virtio/vhost-vdpa-iommufd.c | 240 + > > > > hw/virtio/vhost-vdpa.c | 74 +- > > > > include/hw/virtio/vhost-vdpa.h | 47 +++ > > > > include/hw/virtio/virtio.h | 5 + > > > > linux-headers/linux/vhost.h| 72 ++ > > > > net/vhost-vdpa.c | 31 +++-- > > > > qapi/net.json | 1 + > > > > 8 files changed, 451 insertions(+), 21 deletions(-) > > > > create mode 100644 hw/virtio/vhost-vdpa-iommufd.c > > > > > > > > -- > > > > 2.34.3 > > > > > > > >
[PATCH v16 1/4] vhost: expose function vhost_dev_has_iommu()
To support vIOMMU in vdpa, need to exposed the function
vhost_dev_has_iommu, vdpa will use this function to check
if vIOMMU enable.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 2 +-
include/hw/virtio/vhost.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 746d130c74..23da579ce2 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -107,7 +107,7 @@ static void vhost_dev_sync_region(struct vhost_dev *dev,
}
}
-static bool vhost_dev_has_iommu(struct vhost_dev *dev)
+bool vhost_dev_has_iommu(struct vhost_dev *dev)
{
VirtIODevice *vdev = dev->vdev;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index a52f273347..f7f10c8fb7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -336,4 +336,5 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
struct vhost_inflight *inflight);
int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
struct vhost_inflight *inflight);
+bool vhost_dev_has_iommu(struct vhost_dev *dev);
#endif
--
2.34.3
[PATCH v16 4/4] vhost-vdpa: Add support for vIOMMU.
1. The vIOMMU support will make vDPA can work in IOMMU mode. This
will fix security issues while using the no-IOMMU mode.
To support this feature we need to add new functions for IOMMU MR adds and
deletes.
Also since the SVQ does not support vIOMMU yet, add the check for IOMMU
in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
the function will return fail.
2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While
MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify()
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 145 +++--
include/hw/virtio/vhost-vdpa.h | 11 +++
2 files changed, 149 insertions(+), 7 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 0c8c37e786..b3094e8a8b 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -60,13 +61,21 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, sometimes the section will be larger than iova_max,
+ * but the memory that actually maps is smaller, so move the check to
+ * function vhost_vdpa_iommu_map_notify(). That function will use the
actual
+ * size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
return false;
@@ -185,6 +194,115 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+Int128 llend;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+/* check if RAM section out of device range */
+llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
+if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ v->iova_range.last, int128_get64(llend));
+return;
+}
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+ret = vhost_vdpa_dma_map(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1, vaddr, read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+Int128 end;
+int iommu_idx;
+IOMMUMemoryRegion *iommu_mr;
+i
[PATCH v16 3/4] vhost-vdpa: Add check for full 64-bit in region delete
The unmap ioctl doesn't accept a full 64-bit span. So need to
add check for the section's size in vhost_vdpa_listener_region_del().
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 92c2413c76..0c8c37e786 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -316,10 +316,28 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
vhost_iova_tree_remove(v->iova_tree, *result);
}
vhost_vdpa_iotlb_batch_begin_once(v);
+/*
+ * The unmap ioctl doesn't accept a full 64-bit. need to check it
+ */
+if (int128_eq(llsize, int128_2_64())) {
+llsize = int128_rshift(llsize, 1);
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ int128_get64(llsize));
+
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
+}
+iova += int128_get64(llsize);
+}
ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
+
if (ret) {
-error_report("vhost_vdpa dma unmap error!");
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
}
memory_region_unref(section->mr);
--
2.34.3
[PATCH v16 0/4] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments fix the crash found in test changes in V12 Address the comments, squash patch 1 into the next patch improve the code style issue changes in V13 fail to start if IOMMU and svq enable at same time improve the code style issue changes in V14 Address the comments changes in V15 Address the comments changes in V16 remove the batch mode operation in vIOMMU MR Cindy Lu (4): vhost: expose function vhost_dev_has_iommu() vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del() vhost-vdpa: Add check for full 64-bit in region delete vhost-vdpa: Add support for vIOMMU. hw/virtio/vhost-vdpa.c | 168 +++-- hw/virtio/vhost.c | 2 +- include/hw/virtio/vhost-vdpa.h | 11 +++ include/hw/virtio/vhost.h | 1 + 4 files changed, 172 insertions(+), 10 deletions(-) -- 2.34.3
[PATCH v16 2/4] vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del()
In trace_vhost_vdpa_listener_region_del, the value for llend
should change to int128_get64(int128_sub(llend, int128_one()))
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index bc6bad23d5..92c2413c76 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -288,7 +288,8 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
llend = vhost_vdpa_section_end(section);
-trace_vhost_vdpa_listener_region_del(v, iova, int128_get64(llend));
+trace_vhost_vdpa_listener_region_del(v, iova,
+int128_get64(int128_sub(llend, int128_one(;
if (int128_ge(int128_make64(iova), llend)) {
return;
--
2.34.3
Re: [RFC 0/7] vhost-vdpa: add support for iommufd
On Fri, May 5, 2023 at 11:29 AM Jason Wang wrote: > > Hi Cindy > > On Wed, May 3, 2023 at 5:13 PM Cindy Lu wrote: > > > > Hi All > > There is the RFC to support the IOMMUFD in vdpa device > > any comments are welcome > > Thanks > > Cindy > > Please post the kernel patch as well as a reference. > > Thanks > sure，will do Thanks cindy > > > > Cindy Lu (7): > > vhost: introduce new UAPI to support IOMMUFD > > qapi: support iommufd in vdpa > > virtio : add a ptr for vdpa_iommufd in VirtIODevice > > net/vhost-vdpa: Add the check for iommufd > > vhost-vdpa: Add the iommufd support in the map/unmap function > > vhost-vdpa: init iommufd function in vhost_vdpa start > > vhost-vdpa-iommufd: Add iommufd support for vdpa > > > > hw/virtio/meson.build | 2 +- > > hw/virtio/vhost-vdpa-iommufd.c | 240 + > > hw/virtio/vhost-vdpa.c | 74 +- > > include/hw/virtio/vhost-vdpa.h | 47 +++ > > include/hw/virtio/virtio.h | 5 + > > linux-headers/linux/vhost.h| 72 ++ > > net/vhost-vdpa.c | 31 +++-- > > qapi/net.json | 1 + > > 8 files changed, 451 insertions(+), 21 deletions(-) > > create mode 100644 hw/virtio/vhost-vdpa-iommufd.c > > > > -- > > 2.34.3 > > >
[RFC 5/7] vhost-vdpa: Add the iommufd support in the map/unmap function
1.Change the map/umap function to legacy_map/unmap
2. Add the check for iommufd support,
a>If support iommufd, call the iommufd-related function
b>In order to use kernel's iotlb process. Still need to call
the legacy mode iotlb message, Kernel will check and
skip the legacy iotlb message if iommufd enable
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 56 ++
include/hw/virtio/vhost-vdpa.h | 24 +++
2 files changed, 74 insertions(+), 6 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 542e003101..85240926b2 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "sysemu/iommufd.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -76,8 +77,9 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
* The caller must set asid = 0 if the device does not support asid.
* This is not an ABI break since it is set to 0 by the initializer anyway.
*/
-int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
- hwaddr size, void *vaddr, bool readonly)
+
+int vhost_vdpa_leagcy_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+ hwaddr size, void *vaddr, bool readonly)
{
struct vhost_msg_v2 msg = {};
int fd = v->device_fd;
@@ -103,13 +105,32 @@ int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t
asid, hwaddr iova,
return ret;
}
+int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+ hwaddr size, void *vaddr, bool readonly)
+{
+struct vhost_dev *dev = v->dev;
+
+if ((v->enable_iommufd) && (v->ops == NULL)) {
+vdpa_backend_iommufd_ops_class_init(v);
+}
+
+struct vdpa_iommu_backend_ops *ops = v->ops;
+/* inoder to reuse the iotlb prcess to in kernel, still need to call leagcy
+mode mapping but in kernel , the leagcy mode mapping was replace by
+iommufd*/
+if (v->enable_iommufd) {
+ops->dma_map(dev, asid, iova, size, vaddr, readonly);
+}
+return vhost_vdpa_leagcy_dma_map(v, asid, iova, size, vaddr, readonly);
+}
/*
* The caller must set asid = 0 if the device does not support asid.
* This is not an ABI break since it is set to 0 by the initializer anyway.
*/
-int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
- hwaddr size)
+
+int vhost_vdpa_leagcy_dma_unmap(struct vhost_vdpa *v, uint32_t asid,
+hwaddr iova, hwaddr size)
{
struct vhost_msg_v2 msg = {};
int fd = v->device_fd;
@@ -132,6 +153,26 @@ int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t
asid, hwaddr iova,
return ret;
}
+int vhost_vdpa_dma_unmap(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
+ hwaddr size)
+{
+struct vhost_dev *dev = v->dev;
+
+if ((v->enable_iommufd) && (v->ops == NULL)) {
+vdpa_backend_iommufd_ops_class_init(v);
+}
+
+
+/* inoder to reuse the iotlb prcess to in kernel, still need to call leagcy
+mode mapping but in kernel , the leagcy mode mapping was replace by
+iommufd*/
+if (v->enable_iommufd) {
+struct vdpa_iommu_backend_ops *ops = v->ops;
+
+ops->dma_unmap(dev, asid, iova, size);
+}
+return vhost_vdpa_leagcy_dma_unmap(v, asid, iova, size);
+}
static void vhost_vdpa_listener_begin_batch(struct vhost_vdpa *v)
{
@@ -423,13 +464,14 @@ static void vhost_vdpa_init_svq(struct vhost_dev *hdev,
struct vhost_vdpa *v)
v->shadow_vqs = g_steal_pointer(_vqs);
}
-
+int g_iommufd;
static int vhost_vdpa_init(struct vhost_dev *dev, void *opaque, Error **errp)
{
struct vhost_vdpa *v;
assert(dev->vhost_ops->backend_type == VHOST_BACKEND_TYPE_VDPA);
trace_vhost_vdpa_init(dev, opaque);
int ret;
+printf("[%s] %d called\n", __func__, __LINE__);
/*
* Similar to VFIO, we end up pinning all guest memory and have to
@@ -580,7 +622,9 @@ static int vhost_vdpa_cleanup(struct vhost_dev *dev)
vhost_vdpa_host_notifiers_uninit(dev, dev->nvqs);
memory_listener_unregister(>listener);
vhost_vdpa_svq_cleanup(dev);
-
+if (vhost_vdpa_first_dev(dev)) {
+v->ops->detach_device(v);
+}
dev->opaque = NULL;
ram_block_discard_disable(false);
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 309d4ffc70..aa0e3ed65b 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -55,6 +55,10 @@ typedef struct vhost_vdpa {
void *shadow_vq_ops_opaque;
struct vhost_dev *dev;
VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
+/*
[RFC 2/7] qapi: support iommufd in vdpa
Add a new option for iommufd, The usage is
-object iommufd,id=iommufd0 \
-device virtio-net-pci,netdev=vhost-vdpa1,disable-legacy=on,disable-modern=off\
-netdev
type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa1,iommufd=iommufd0\
...
Signed-off-by: Cindy Lu
---
qapi/net.json | 1 +
1 file changed, 1 insertion(+)
diff --git a/qapi/net.json b/qapi/net.json
index 522ac582ed..fffaf9bb5e 100644
--- a/qapi/net.json
+++ b/qapi/net.json
@@ -461,6 +461,7 @@
'*vhostdev': 'str',
'*vhostfd': 'str',
'*queues': 'int',
+'*iommufd': 'str',
'*x-svq':{'type': 'bool', 'features' : [ 'unstable'] } } }
##
--
2.34.3
[RFC 7/7] vhost-vdpa-iommufd: Add iommufd support for vdpa
This file is support iommufd for vdpa, including the function:
1> iommufd bind/unbind the iommufd device
bind the vdpa device to iommufd and attach the ASID 0 to iommufd
2> iommufd map/unmap function.The map function working process is
a. Check if the asid was used before.
b. If this is the new asid, get the new ioas_id and attach it to iommufd.
save this information in vdpa_iommufd.
c. Use the ioas_id for mapping
The unmap logic is the same
Signed-off-by: Cindy Lu
---
hw/virtio/meson.build | 2 +-
hw/virtio/vhost-vdpa-iommufd.c | 240 +
2 files changed, 241 insertions(+), 1 deletion(-)
create mode 100644 hw/virtio/vhost-vdpa-iommufd.c
diff --git a/hw/virtio/meson.build b/hw/virtio/meson.build
index f93be2e137..848fdb18eb 100644
--- a/hw/virtio/meson.build
+++ b/hw/virtio/meson.build
@@ -13,7 +13,7 @@ if have_vhost
specific_virtio_ss.add(files('vhost-user.c'))
endif
if have_vhost_vdpa
-specific_virtio_ss.add(files('vhost-vdpa.c', 'vhost-shadow-virtqueue.c'))
+specific_virtio_ss.add(files('vhost-vdpa.c',
'vhost-shadow-virtqueue.c','vhost-vdpa-iommufd.c'))
endif
else
softmmu_virtio_ss.add(files('vhost-stub.c'))
diff --git a/hw/virtio/vhost-vdpa-iommufd.c b/hw/virtio/vhost-vdpa-iommufd.c
new file mode 100644
index 00..6a0875c0a4
--- /dev/null
+++ b/hw/virtio/vhost-vdpa-iommufd.c
@@ -0,0 +1,240 @@
+
+#include "qemu/osdep.h"
+#include
+#include
+#include
+#include
+#include "sysemu/iommufd.h"
+#include "hw/virtio/vhost.h"
+
+#include "hw/virtio/vhost-vdpa.h"
+
+static int vdpa_device_attach_ioas(struct vhost_vdpa *dev,
+ VDPAIOMMUFDState *vdpa_iommufd)
+{
+int ret;
+
+struct vdpa_device_attach_iommufd_as attach_data = {
+.argsz = sizeof(attach_data),
+.flags = 0,
+.ioas_id = vdpa_iommufd->ioas_id,
+};
+/* Attach device to an ioas within iommufd */
+ret = ioctl(dev->device_fd, VDPA_DEVICE_ATTACH_IOMMUFD_AS, _data);
+if (ret) {
+error_report("fail to bind device fd=%d to ioas_id=%d", dev->device_fd,
+ vdpa_iommufd->ioas_id);
+return ret;
+}
+
+return 0;
+}
+static VDPAIOMMUFDState *vdpa_get_ioas_by_asid(struct vhost_dev *hdev,
+ uint32_t asid)
+{
+VDPAIOMMUFDState *vdpa_iommufd_ptr = hdev->vdev->iommufd_ptr;
+while (vdpa_iommufd_ptr != NULL) {
+if (asid == vdpa_iommufd_ptr->asid) {
+return vdpa_iommufd_ptr;
+}
+
+vdpa_iommufd_ptr = vdpa_iommufd_ptr->next;
+}
+
+return NULL;
+}
+static VDPAIOMMUFDState *vdpa_add_new_ioas_id(struct vhost_dev *hdev,
+ uint32_t asid)
+{
+int ret;
+uint32_t ioas_id;
+
+struct vhost_vdpa *v = hdev->opaque;
+VDPAIOMMUFDState *vdpa_iommufd_ptr = hdev->vdev->iommufd_ptr;
+VDPAIOMMUFDState *vdpa_iommufd_new = g_malloc(sizeof(VDPAIOMMUFDState));
+
+vdpa_iommufd_new->dev = hdev;
+vdpa_iommufd_new->asid = asid;
+vdpa_iommufd_new->iommufd = vdpa_iommufd_ptr->iommufd;
+
+ret = iommufd_backend_get_ioas(vdpa_iommufd_new->iommufd, _id);
+if (ret < 0) {
+error_report("Failed to alloc ioas (%s)", strerror(errno));
+return NULL;
+}
+
+vdpa_iommufd_new->ioas_id = ioas_id;
+/* this is new asid, attch to iommufd*/
+ret = vdpa_device_attach_ioas(v, vdpa_iommufd_new);
+if (ret < 0) {
+error_report("Failed to attach ioas (%s)", strerror(errno));
+return NULL;
+}
+while (vdpa_iommufd_ptr->next != NULL) {
+vdpa_iommufd_ptr = vdpa_iommufd_ptr->next;
+}
+/*save this vdpa_iommufd in list */
+vdpa_iommufd_ptr->next = vdpa_iommufd_new;
+vdpa_iommufd_new->next = NULL;
+return vdpa_iommufd_new;
+}
+static int vdpa_iommufd_map(struct vhost_dev *hdev, uint32_t asid, hwaddr iova,
+hwaddr size, void *vaddr, bool readonly)
+{
+VDPAIOMMUFDState *vdpa_iommufd;
+
+if (hdev->vdev == NULL) {
+error_report("Failed to get vdev (%s)", strerror(errno));
+return 0;
+}
+/*search if this asid have attach to iommufd before*/
+vdpa_iommufd = vdpa_get_ioas_by_asid(hdev, asid);
+if (vdpa_iommufd == NULL) {
+/*this asid is first use, need to alloc and add to iommufd*/
+vdpa_iommufd = vdpa_add_new_ioas_id(hdev, asid);
+}
+return iommufd_backend_map_dma(vdpa_iommufd->iommufd,
vdpa_iommufd->ioas_id,
+ iova, size, vaddr, readonly);
+}
+
+
+static int vdpa_iommufd_unmap(struct vhost_dev *hdev, uint32_t asid,
+ hwaddr iova, hwaddr size)
+{
+VDPAIOMMUFDState *vdpa_iommufd;
+if (hdev->vdev == NULL) {
+
[RFC 1/7] vhost: introduce new UAPI to support IOMMUFD
Add 3 new UAPI
VHOST_VDPA_SET_IOMMU_FD: this to bind the vdpa device to iommufd
VDPA_DEVICE_ATTACH_IOMMUFD_AS: attach new ioas to iommufd
VDPA_DEVICE_DTTACH_IOMMUFD_AS: detach all the ioas from iommufd
Signed-off-by: Cindy Lu
---
linux-headers/linux/vhost.h | 72 +
1 file changed, 72 insertions(+)
diff --git a/linux-headers/linux/vhost.h b/linux-headers/linux/vhost.h
index f9f115a7c7..bf426177f3 100644
--- a/linux-headers/linux/vhost.h
+++ b/linux-headers/linux/vhost.h
@@ -180,4 +180,76 @@
*/
#define VHOST_VDPA_SUSPEND _IO(VHOST_VIRTIO, 0x7D)
+/* vhost vdpa set iommufd
+ * Input parameters:
+ * @iommufd: file descriptor from /dev/iommu; pass -1 to unset
+ * @group_id: identifier of the group that a virtqueue belongs to
+ * @ioas_id: IOAS identifier returned from ioctl(IOMMU_IOAS_ALLOC)
+ * Output parameters:
+ * @out_dev_id: device identifier
+ * @out_hwpt_id: hardware IO pagetable identifier
+ */
+struct vhost_vdpa_set_iommufd {
+__s32 iommufd;
+__u32 group_id;
+__u32 ioas_id;
+__u32 out_devid;
+__u32 out_hwptid;
+};
+
+#define VHOST_VDPA_SET_IOMMU_FD \
+_IOW(VHOST_VIRTIO, 0x7e, struct vhost_vdpa_set_iommufd)
+
+/*
+ * VDPA_DEVICE_ATTACH_IOMMUFD_AS -
+ * _IOW(VHOST_VIRTIO, 0x7f, struct vdpa_device_attach_iommufd_as)
+ *
+ * Attach a vdpa device to an iommufd address space specified by IOAS
+ * id.
+ *
+ * Available only after a device has been bound to iommufd via
+ * VHOST_VDPA_SET_IOMMU_FD
+ *
+ * Undo by VDPA_DEVICE_DETACH_IOMMUFD_AS or device fd close.
+ *
+ * @argsz: user filled size of this data.
+ * @flags: must be 0.
+ * @ioas_id: Input the target id which can represent an ioas
+ * allocated via iommufd subsystem.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+struct vdpa_device_attach_iommufd_as {
+__u32 argsz;
+__u32 flags;
+__u32 ioas_id;
+};
+
+#define VDPA_DEVICE_ATTACH_IOMMUFD_AS \
+_IOW(VHOST_VIRTIO, 0x7f, struct vdpa_device_attach_iommufd_as)
+
+
+/*
+ * VDPA_DEVICE_DETACH_IOMMUFD_AS
+ *
+ * Detach a vdpa device from the iommufd address space it has been
+ * attached to. After it, device should be in a blocking DMA state.
+ *
+ * Available only after a device has been bound to iommufd via
+ * VHOST_VDPA_SET_IOMMU_FD
+ *
+ * @argsz: user filled size of this data.
+ * @flags: must be 0.
+ *
+ * Return: 0 on success, -errno on failure.
+ */
+struct vdpa_device_detach_iommufd_as {
+__u32 argsz;
+__u32 flags;
+};
+
+#define VDPA_DEVICE_DETACH_IOMMUFD_AS \
+_IOW(VHOST_VIRTIO, 0x83, struct vdpa_device_detach_iommufd_as)
+
+
#endif
--
2.34.3
[RFC 4/7] net/vhost-vdpa: Add the check for iommufd
Add the check for object iommufd, if the iommfd enabled.
pass the information to vhost_vdpa.
vhost_vdpa dev start will check this bool and connect
to the iommufd
Signed-off-by: Cindy Lu
---
net/vhost-vdpa.c | 31 +--
1 file changed, 17 insertions(+), 14 deletions(-)
diff --git a/net/vhost-vdpa.c b/net/vhost-vdpa.c
index 1a13a34d35..d4819c28e1 100644
--- a/net/vhost-vdpa.c
+++ b/net/vhost-vdpa.c
@@ -659,16 +659,12 @@ static const VhostShadowVirtqueueOps
vhost_vdpa_net_svq_ops = {
.avail_handler = vhost_vdpa_net_handle_ctrl_avail,
};
-static NetClientState *net_vhost_vdpa_init(NetClientState *peer,
- const char *device,
- const char *name,
- int vdpa_device_fd,
- int queue_pair_index,
- int nvqs,
- bool is_datapath,
- bool svq,
- struct vhost_vdpa_iova_range iova_range,
- VhostIOVATree *iova_tree)
+static NetClientState *
+net_vhost_vdpa_init(NetClientState *peer, const char *device, const char *name,
+int vdpa_device_fd, int queue_pair_index, int nvqs,
+bool is_datapath, bool svq, bool enable_iommufd,
+struct vhost_vdpa_iova_range iova_range,
+VhostIOVATree *iova_tree)
{
NetClientState *nc = NULL;
VhostVDPAState *s;
@@ -691,6 +687,7 @@ static NetClientState *net_vhost_vdpa_init(NetClientState
*peer,
s->vhost_vdpa.iova_range = iova_range;
s->vhost_vdpa.shadow_data = svq;
s->vhost_vdpa.iova_tree = iova_tree;
+s->vhost_vdpa.enable_iommufd = enable_iommufd;
if (!is_datapath) {
s->cvq_cmd_out_buffer = qemu_memalign(qemu_real_host_page_size(),
vhost_vdpa_net_cvq_cmd_page_len());
@@ -793,6 +790,12 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char
*name,
}
}
+bool enable_iommufd = false;
+if (opts->iommufd) {
+enable_iommufd = true;
+printf("[%s] %d called\n", __func__, __LINE__);
+}
+
r = vhost_vdpa_get_features(vdpa_device_fd, , errp);
if (unlikely(r < 0)) {
goto err;
@@ -825,15 +828,15 @@ int net_init_vhost_vdpa(const Netdev *netdev, const char
*name,
for (i = 0; i < queue_pairs; i++) {
ncs[i] = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
vdpa_device_fd, i, 2, true, opts->x_svq,
- iova_range, iova_tree);
+ enable_iommufd, iova_range, iova_tree);
if (!ncs[i])
goto err;
}
if (has_cvq) {
-nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name,
- vdpa_device_fd, i, 1, false,
- opts->x_svq, iova_range, iova_tree);
+nc = net_vhost_vdpa_init(peer, TYPE_VHOST_VDPA, name, vdpa_device_fd,
i,
+ 1, false, opts->x_svq, enable_iommufd,
+ iova_range, iova_tree);
if (!nc)
goto err;
}
--
2.34.3
[RFC 6/7] vhost-vdpa: init iommufd function in vhost_vdpa start
Add support for iommufd, init the vdpa_iommufd in vdpa_start
in this step, driver will bind to the iommufd device
and attach the default ASID(asid 0) to iommufd
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 18 ++
1 file changed, 18 insertions(+)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 85240926b2..6c01e3b44f 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -1158,6 +1158,24 @@ static int vhost_vdpa_dev_start(struct vhost_dev *dev,
bool started)
trace_vhost_vdpa_dev_start(dev, started);
if (started) {
+if ((v->enable_iommufd) && (vhost_vdpa_first_dev(dev))) {
+struct vdpa_iommufd *vdpa_iommufd;
+
+vdpa_backend_iommufd_ops_class_init(v);
+
+if (dev->vdev->iommufd_ptr == NULL) {
+vdpa_iommufd = g_malloc(sizeof(VDPAIOMMUFDState));
+
+vdpa_iommufd->iommufd = g_malloc(sizeof(IOMMUFDBackend));
+dev->vdev->iommufd_ptr = vdpa_iommufd;
+
+qemu_mutex_init(_iommufd->iommufd->lock);
+iommufd_backend_connect(vdpa_iommufd->iommufd, NULL);
+
+v->ops->attach_device(v, dev->vdev->dma_as, NULL);
+}
+}
+
vhost_vdpa_host_notifiers_init(dev);
ok = vhost_vdpa_svqs_start(dev);
if (unlikely(!ok)) {
--
2.34.3
[RFC 0/7] vhost-vdpa: add support for iommufd
Hi All There is the RFC to support the IOMMUFD in vdpa device any comments are welcome Thanks Cindy Cindy Lu (7): vhost: introduce new UAPI to support IOMMUFD qapi: support iommufd in vdpa virtio : add a ptr for vdpa_iommufd in VirtIODevice net/vhost-vdpa: Add the check for iommufd vhost-vdpa: Add the iommufd support in the map/unmap function vhost-vdpa: init iommufd function in vhost_vdpa start vhost-vdpa-iommufd: Add iommufd support for vdpa hw/virtio/meson.build | 2 +- hw/virtio/vhost-vdpa-iommufd.c | 240 + hw/virtio/vhost-vdpa.c | 74 +- include/hw/virtio/vhost-vdpa.h | 47 +++ include/hw/virtio/virtio.h | 5 + linux-headers/linux/vhost.h| 72 ++ net/vhost-vdpa.c | 31 +++-- qapi/net.json | 1 + 8 files changed, 451 insertions(+), 21 deletions(-) create mode 100644 hw/virtio/vhost-vdpa-iommufd.c -- 2.34.3
[RFC 3/7] virtio : add a ptr for vdpa_iommufd in VirtIODevice
To support iommufd, vdpa needs to save the ioas_id and the ASID,
which need to be shared between all vhost_vdpa devices.
So Add a pointer in VirtIODevice.
vdpa device need to init it when the dev start, Add all the vdpa device
will read/write this same ptr. TODO: need to add a lock for
read and write
Signed-off-by: Cindy Lu
---
include/hw/virtio/vhost-vdpa.h | 23 +++
include/hw/virtio/virtio.h | 5 +
2 files changed, 28 insertions(+)
diff --git a/include/hw/virtio/vhost-vdpa.h b/include/hw/virtio/vhost-vdpa.h
index 7997f09a8d..309d4ffc70 100644
--- a/include/hw/virtio/vhost-vdpa.h
+++ b/include/hw/virtio/vhost-vdpa.h
@@ -18,6 +18,10 @@
#include "hw/virtio/vhost-shadow-virtqueue.h"
#include "hw/virtio/virtio.h"
#include "standard-headers/linux/vhost_types.h"
+//#include "sysemu/iommufd.h"
+#include "qemu/osdep.h"
+#include "sysemu/sysemu.h"
+
/*
* ASID dedicated to map guest's addresses. If SVQ is disabled it maps GPA to
@@ -30,6 +34,8 @@ typedef struct VhostVDPAHostNotifier {
void *addr;
} VhostVDPAHostNotifier;
+typedef struct IOMMUFDBackend IOMMUFDBackend;
+
typedef struct vhost_vdpa {
int device_fd;
int index;
@@ -51,6 +57,23 @@ typedef struct vhost_vdpa {
VhostVDPAHostNotifier notifier[VIRTIO_QUEUE_MAX];
} VhostVDPA;
+
+typedef struct vdpa_iommufd {
+IOMMUFDBackend *iommufd;
+struct vhost_dev *dev;
+/*ioas_id get from IOMMUFD, iommufd need to use this id to map,unmap*/
+uint32_t ioas_id;
+/*ASID used for vq*/
+uint32_t asid;
+__u32 devid; /*not use */
+__u32 hwptid; /*not use*/
+AddressSpace *as;
+struct vdpa_iommufd *next;
+// QLIST_ENTRY(vdpa_iommufd) iommufd_next;
+
+} VDPAIOMMUFDState;
+
+
int vhost_vdpa_get_iova_range(int fd, struct vhost_vdpa_iova_range
*iova_range);
int vhost_vdpa_dma_map(struct vhost_vdpa *v, uint32_t asid, hwaddr iova,
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 77c6c55929..36b4783466 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -152,6 +152,11 @@ struct VirtIODevice
uint8_t device_endian;
bool use_guest_notifier_mask;
AddressSpace *dma_as;
+/*this is an ptr point to struct vdpa_iommufd, will change to QLIST if
+ * needed*/
+/*in this struct saved the ioas_id/ASID that we need to use in iommufd
+map/unmap, this ioas_id/ASID will share between vqs,so we add the ptr
here*/
+void *iommufd_ptr;
QLIST_HEAD(, VirtQueue) *vector_queues;
QTAILQ_ENTRY(VirtIODevice) next;
EventNotifier config_notifier;
--
2.34.3
Re: [PATCH v15 4/4] vhost-vdpa: Add support for vIOMMU.
On Fri, Mar 24, 2023 at 10:49 AM Jason Wang wrote: > > On Thu, Mar 23, 2023 at 4:41 PM Cindy Lu wrote: > > > > On Thu, Mar 23, 2023 at 11:47 AM Jason Wang wrote: > > > > > > On Tue, Mar 21, 2023 at 10:24 PM Cindy Lu wrote: > > > > > > > > 1. The vIOMMU support will make vDPA can work in IOMMU mode. This > > > > will fix security issues while using the no-IOMMU mode. > > > > To support this feature we need to add new functions for IOMMU MR adds > > > > and > > > > deletes. > > > > > > > > Also since the SVQ does not support vIOMMU yet, add the check for IOMMU > > > > in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time > > > > the function will return fail. > > > > > > > > 2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While > > > > MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify() > > > > > > > > Verified in vp_vdpa and vdpa_sim_net driver > > > > > > > > Signed-off-by: Cindy Lu > > > > --- > > > > hw/virtio/trace-events | 2 +- > > > > hw/virtio/vhost-vdpa.c | 159 ++--- > > > > include/hw/virtio/vhost-vdpa.h | 11 +++ > > > > 3 files changed, 161 insertions(+), 11 deletions(-) > > > > > > > > diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events > > > > index 8f8d05cf9b..de4da2c65c 100644 > > > > --- a/hw/virtio/trace-events > > > > +++ b/hw/virtio/trace-events > > > > @@ -33,7 +33,7 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d > > > > n:%p" > > > > vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t > > > > asid, uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, > > > > uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: > > > > 0x%"PRIx64" size: 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: > > > > %"PRIu8 > > > > vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t > > > > asid, uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d > > > > msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" > > > > type: %"PRIu8 > > > > vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, > > > > uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 > > > > -vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t > > > > type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 > > > > +vhost_vdpa_iotlb_batch_end_once(void *v, int fd, uint32_t msg_type, > > > > uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8 > > > > vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t > > > > llend, void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend > > > > 0x%"PRIx64" vaddr: %p read-only: %d" > > > > vhost_vdpa_listener_region_del(void *vdpa, uint64_t iova, uint64_t > > > > llend) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64 > > > > vhost_vdpa_add_status(void *dev, uint8_t status) "dev: %p status: > > > > 0x%"PRIx8 > > > > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c > > > > index 0c8c37e786..39720d12a6 100644 > > > > --- a/hw/virtio/vhost-vdpa.c > > > > +++ b/hw/virtio/vhost-vdpa.c > > > > @@ -26,6 +26,7 @@ > > > > #include "cpu.h" > > > > #include "trace.h" > > > > #include "qapi/error.h" > > > > +#include "hw/virtio/virtio-access.h" > > > > > > > > /* > > > > * Return one past the end of the end of section. Be careful with > > > > uint64_t > > > > @@ -60,13 +61,21 @@ static bool > > > > vhost_vdpa_listener_skipped_section(MemoryRegionSection *section, > > > > iova_min, section->offset_within_address_space); > > > > return true; > > > > } > > > > +/* > > > > + * While using vIOMMU, sometimes the section will be larger than > > > > iova_max, > > > > + * but the memo
Re: [PATCH v15 4/4] vhost-vdpa: Add support for vIOMMU.
On Thu, Mar 23, 2023 at 11:47 AM Jason Wang wrote:
>
> On Tue, Mar 21, 2023 at 10:24 PM Cindy Lu wrote:
> >
> > 1. The vIOMMU support will make vDPA can work in IOMMU mode. This
> > will fix security issues while using the no-IOMMU mode.
> > To support this feature we need to add new functions for IOMMU MR adds and
> > deletes.
> >
> > Also since the SVQ does not support vIOMMU yet, add the check for IOMMU
> > in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
> > the function will return fail.
> >
> > 2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While
> > MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify()
> >
> > Verified in vp_vdpa and vdpa_sim_net driver
> >
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/trace-events | 2 +-
> > hw/virtio/vhost-vdpa.c | 159 ++---
> > include/hw/virtio/vhost-vdpa.h | 11 +++
> > 3 files changed, 161 insertions(+), 11 deletions(-)
> >
> > diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
> > index 8f8d05cf9b..de4da2c65c 100644
> > --- a/hw/virtio/trace-events
> > +++ b/hw/virtio/trace-events
> > @@ -33,7 +33,7 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
> > vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid,
> > uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type)
> > "vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size:
> > 0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
> > vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid,
> > uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type:
> > %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
> > vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type,
> > uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
> > -vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t
> > type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
> > +vhost_vdpa_iotlb_batch_end_once(void *v, int fd, uint32_t msg_type,
> > uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
> > vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend,
> > void *vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64"
> > vaddr: %p read-only: %d"
> > vhost_vdpa_listener_region_del(void *vdpa, uint64_t iova, uint64_t llend)
> > "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64
> > vhost_vdpa_add_status(void *dev, uint8_t status) "dev: %p status: 0x%"PRIx8
> > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> > index 0c8c37e786..39720d12a6 100644
> > --- a/hw/virtio/vhost-vdpa.c
> > +++ b/hw/virtio/vhost-vdpa.c
> > @@ -26,6 +26,7 @@
> > #include "cpu.h"
> > #include "trace.h"
> > #include "qapi/error.h"
> > +#include "hw/virtio/virtio-access.h"
> >
> > /*
> > * Return one past the end of the end of section. Be careful with uint64_t
> > @@ -60,13 +61,21 @@ static bool
> > vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
> > iova_min, section->offset_within_address_space);
> > return true;
> > }
> > +/*
> > + * While using vIOMMU, sometimes the section will be larger than
> > iova_max,
> > + * but the memory that actually maps is smaller, so move the check to
> > + * function vhost_vdpa_iommu_map_notify(). That function will use the
> > actual
> > + * size that maps to the kernel
> > + */
> >
> > -llend = vhost_vdpa_section_end(section);
> > -if (int128_gt(llend, int128_make64(iova_max))) {
> > -error_report("RAM section out of device range (max=0x%" PRIx64
> > - ", end addr=0x%" PRIx64 ")",
> > - iova_max, int128_get64(llend));
> > -return true;
> > +if (!memory_region_is_iommu(section->mr)) {
> > +llend = vhost_vdpa_section_end(section);
> > +if (int128_gt(llend, int128_make64(iova_max))) {
> > +error_report("RAM section out of device range (max=0x%" PRIx64
> > +
[PATCH v15 2/4] vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del()
In trace_vhost_vdpa_listener_region_del, the value for llend
should change to int128_get64(int128_sub(llend, int128_one()))
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index bc6bad23d5..92c2413c76 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -288,7 +288,8 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
llend = vhost_vdpa_section_end(section);
-trace_vhost_vdpa_listener_region_del(v, iova, int128_get64(llend));
+trace_vhost_vdpa_listener_region_del(v, iova,
+int128_get64(int128_sub(llend, int128_one(;
if (int128_ge(int128_make64(iova), llend)) {
return;
--
2.34.3
[PATCH v15 4/4] vhost-vdpa: Add support for vIOMMU.
1. The vIOMMU support will make vDPA can work in IOMMU mode. This
will fix security issues while using the no-IOMMU mode.
To support this feature we need to add new functions for IOMMU MR adds and
deletes.
Also since the SVQ does not support vIOMMU yet, add the check for IOMMU
in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
the function will return fail.
2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While
MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify()
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/trace-events | 2 +-
hw/virtio/vhost-vdpa.c | 159 ++---
include/hw/virtio/vhost-vdpa.h | 11 +++
3 files changed, 161 insertions(+), 11 deletions(-)
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index 8f8d05cf9b..de4da2c65c 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -33,7 +33,7 @@ vhost_user_create_notifier(int idx, void *n) "idx:%d n:%p"
vhost_vdpa_dma_map(void *vdpa, int fd, uint32_t msg_type, uint32_t asid,
uint64_t iova, uint64_t size, uint64_t uaddr, uint8_t perm, uint8_t type)
"vdpa:%p fd: %d msg_type: %"PRIu32" asid: %"PRIu32" iova: 0x%"PRIx64" size:
0x%"PRIx64" uaddr: 0x%"PRIx64" perm: 0x%"PRIx8" type: %"PRIu8
vhost_vdpa_dma_unmap(void *vdpa, int fd, uint32_t msg_type, uint32_t asid,
uint64_t iova, uint64_t size, uint8_t type) "vdpa:%p fd: %d msg_type: %"PRIu32"
asid: %"PRIu32" iova: 0x%"PRIx64" size: 0x%"PRIx64" type: %"PRIu8
vhost_vdpa_listener_begin_batch(void *v, int fd, uint32_t msg_type, uint8_t
type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
-vhost_vdpa_listener_commit(void *v, int fd, uint32_t msg_type, uint8_t type)
"vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
+vhost_vdpa_iotlb_batch_end_once(void *v, int fd, uint32_t msg_type, uint8_t
type) "vdpa:%p fd: %d msg_type: %"PRIu32" type: %"PRIu8
vhost_vdpa_listener_region_add(void *vdpa, uint64_t iova, uint64_t llend, void
*vaddr, bool readonly) "vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64" vaddr: %p
read-only: %d"
vhost_vdpa_listener_region_del(void *vdpa, uint64_t iova, uint64_t llend)
"vdpa: %p iova 0x%"PRIx64" llend 0x%"PRIx64
vhost_vdpa_add_status(void *dev, uint8_t status) "dev: %p status: 0x%"PRIx8
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 0c8c37e786..39720d12a6 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -60,13 +61,21 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, sometimes the section will be larger than iova_max,
+ * but the memory that actually maps is smaller, so move the check to
+ * function vhost_vdpa_iommu_map_notify(). That function will use the
actual
+ * size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
return false;
@@ -158,9 +167,8 @@ static void vhost_vdpa_iotlb_batch_begin_once(struct
vhost_vdpa *v)
v->iotlb_batch_begin_sent = true;
}
-static void vhost_vdpa_listener_commit(MemoryListener *listener)
+static void vhost_vdpa_iotlb_batch_end_once(struct vhost_vdpa *v)
{
-struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
struct vhost_dev *dev = v->dev;
struct vhost_msg_v2 msg = {};
int fd = v->device_fd;
@@ -176,7 +184,7 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
msg.type = v->msg_type;
msg.iotlb.type = VHOST_IOTLB_BATCH_END;
-trace_vhost_vdpa_listener_commit(v, fd, msg.type, msg.iotlb.type);
+trace_vhost_vdpa_iotlb_batch_end_once(v, fd, msg.type, ms
[PATCH v15 1/4] vhost: expose function vhost_dev_has_iommu()
To support vIOMMU in vdpa, need to exposed the function
vhost_dev_has_iommu, vdpa will use this function to check
if vIOMMU enable.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 2 +-
include/hw/virtio/vhost.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index a266396576..fd746b085b 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -107,7 +107,7 @@ static void vhost_dev_sync_region(struct vhost_dev *dev,
}
}
-static bool vhost_dev_has_iommu(struct vhost_dev *dev)
+bool vhost_dev_has_iommu(struct vhost_dev *dev)
{
VirtIODevice *vdev = dev->vdev;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index a52f273347..f7f10c8fb7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -336,4 +336,5 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
struct vhost_inflight *inflight);
int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
struct vhost_inflight *inflight);
+bool vhost_dev_has_iommu(struct vhost_dev *dev);
#endif
--
2.34.3
[PATCH v15 3/4] vhost-vdpa: Add check for full 64-bit in region delete
The unmap ioctl doesn't accept a full 64-bit span. So need to
add check for the section's size in vhost_vdpa_listener_region_del().
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 92c2413c76..0c8c37e786 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -316,10 +316,28 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
vhost_iova_tree_remove(v->iova_tree, *result);
}
vhost_vdpa_iotlb_batch_begin_once(v);
+/*
+ * The unmap ioctl doesn't accept a full 64-bit. need to check it
+ */
+if (int128_eq(llsize, int128_2_64())) {
+llsize = int128_rshift(llsize, 1);
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ int128_get64(llsize));
+
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
+}
+iova += int128_get64(llsize);
+}
ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
+
if (ret) {
-error_report("vhost_vdpa dma unmap error!");
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
}
memory_region_unref(section->mr);
--
2.34.3
[PATCH v15 0/4] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments fix the crash found in test changes in V12 Address the comments, squash patch 1 into the next patch improve the code style issue changes in V13 fail to start if IOMMU and svq enable at same time improve the code style issue changes in V14 Address the comments changes in V15 Address the comments Cindy Lu (4): vhost: expose function vhost_dev_has_iommu() vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del() vhost-vdpa: Add check for full 64-bit in region delete vhost-vdpa: Add support for vIOMMU. hw/virtio/trace-events | 2 +- hw/virtio/vhost-vdpa.c | 182 ++--- hw/virtio/vhost.c | 2 +- include/hw/virtio/vhost-vdpa.h | 11 ++ include/hw/virtio/vhost.h | 1 + 5 files changed, 184 insertions(+), 14 deletions(-) -- 2.34.3
Re: [PATCH v14 4/4] vhost-vdpa: Add support for vIOMMU.
On Tue, Mar 21, 2023 at 11:21 AM Jason Wang wrote:
>
> On Tue, Mar 21, 2023 at 12:20 AM Cindy Lu wrote:
> >
> > 1. The vIOMMU support will make vDPA can work in IOMMU mode. This
> > will fix security issues while using the no-IOMMU mode.
> > To support this feature we need to add new functions for IOMMU MR adds and
> > deletes.
> >
> > Also since the SVQ does not support vIOMMU yet, add the check for IOMMU
> > in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
> > the function will return fail.
> >
> > 2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While
> > MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify()
> >
> > Verified in vp_vdpa and vdpa_sim_net driver
> >
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/vhost-vdpa.c | 149 +++--
> > include/hw/virtio/vhost-vdpa.h | 11 +++
> > 2 files changed, 152 insertions(+), 8 deletions(-)
> >
> > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> > index 0c8c37e786..b36922b365 100644
> > --- a/hw/virtio/vhost-vdpa.c
> > +++ b/hw/virtio/vhost-vdpa.c
> > @@ -26,6 +26,7 @@
> > #include "cpu.h"
> > #include "trace.h"
> > #include "qapi/error.h"
> > +#include "hw/virtio/virtio-access.h"
> >
> > /*
> > * Return one past the end of the end of section. Be careful with uint64_t
> > @@ -60,15 +61,22 @@ static bool
> > vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
> > iova_min, section->offset_within_address_space);
> > return true;
> > }
> > +/*
> > + * While using vIOMMU, sometimes the section will be larger than
> > iova_max,
> > + * but the memory that actually maps is smaller, so move the check to
> > + * function vhost_vdpa_iommu_map_notify(). That function will use the
> > actual
> > + * size that maps to the kernel
> > + */
> >
> > -llend = vhost_vdpa_section_end(section);
> > -if (int128_gt(llend, int128_make64(iova_max))) {
> > -error_report("RAM section out of device range (max=0x%" PRIx64
> > - ", end addr=0x%" PRIx64 ")",
> > - iova_max, int128_get64(llend));
> > -return true;
> > +if (!memory_region_is_iommu(section->mr)) {
> > +llend = vhost_vdpa_section_end(section);
> > +if (int128_gt(llend, int128_make64(iova_max))) {
> > +error_report("RAM section out of device range (max=0x%" PRIx64
> > + ", end addr=0x%" PRIx64 ")",
> > + iova_max, int128_get64(llend));
> > +return true;
> > +}
> > }
> > -
>
> Unnecessary changes.
>
will fix this
> > return false;
> > }
> >
> > @@ -185,6 +193,118 @@ static void vhost_vdpa_listener_commit(MemoryListener
> > *listener)
> > v->iotlb_batch_begin_sent = false;
> > }
> >
> > +static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry
> > *iotlb)
> > +{
> > +struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
> > +
> > +hwaddr iova = iotlb->iova + iommu->iommu_offset;
> > +struct vhost_vdpa *v = iommu->dev;
> > +void *vaddr;
> > +int ret;
> > +Int128 llend;
> > +
> > +if (iotlb->target_as != _space_memory) {
> > +error_report("Wrong target AS \"%s\", only system memory is
> > allowed",
> > + iotlb->target_as->name ? iotlb->target_as->name :
> > "none");
> > +return;
> > +}
> > +RCU_READ_LOCK_GUARD();
> > +/* check if RAM section out of device range */
> > +llend = int128_add(int128_makes64(iotlb->addr_mask),
> > int128_makes64(iova));
> > +if (int128_gt(llend, int128_make64(v->iova_range.last))) {
> > +error_report("RAM section out of device range (max=0x%" PRIx64
> > + ", end addr=0x%" PRIx64 ")",
> > + v->iova_range.last, int128_get64(llend));
> > +return;
> > +}
> > +
> > +vhost_vdpa_iotlb_batch_begin_once(v);
>
> Quoted from you answer in V1:
>
> "
> the VHOST_IOTLB_BATCH_END message was send by
> vhost_vdpa_listener_commit, because we only use
> one vhost_vdpa_memory_listener and no-iommu mode will also need to use
> this listener, So we still need to add the batch begin here, based on
> my testing after the notify function was called, the listener_commit
> function was also called .so it works well in this situation
> "
>
> This assumes the map_notify to be called within the memory
> transactions which is not necessarily the case.
>
> I think it could be triggered when guest tries to establish a new
> mapping in the vIOMMU. In this case there's no memory transactions at
> all?
>
sure, thanks will fix this
> Thanks
>
[PATCH v14 1/4] vhost: expose function vhost_dev_has_iommu()
To support vIOMMU in vdpa, need to exposed the function
vhost_dev_has_iommu, vdpa will use this function to check
if vIOMMU enable.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 2 +-
include/hw/virtio/vhost.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index a266396576..fd746b085b 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -107,7 +107,7 @@ static void vhost_dev_sync_region(struct vhost_dev *dev,
}
}
-static bool vhost_dev_has_iommu(struct vhost_dev *dev)
+bool vhost_dev_has_iommu(struct vhost_dev *dev)
{
VirtIODevice *vdev = dev->vdev;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index a52f273347..f7f10c8fb7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -336,4 +336,5 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
struct vhost_inflight *inflight);
int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
struct vhost_inflight *inflight);
+bool vhost_dev_has_iommu(struct vhost_dev *dev);
#endif
--
2.34.3
[PATCH v14 2/4] vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del()
In trace_vhost_vdpa_listener_region_del, the value for llend
should change to int128_get64(int128_sub(llend, int128_one()))
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index bc6bad23d5..92c2413c76 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -288,7 +288,8 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
iova = TARGET_PAGE_ALIGN(section->offset_within_address_space);
llend = vhost_vdpa_section_end(section);
-trace_vhost_vdpa_listener_region_del(v, iova, int128_get64(llend));
+trace_vhost_vdpa_listener_region_del(v, iova,
+int128_get64(int128_sub(llend, int128_one(;
if (int128_ge(int128_make64(iova), llend)) {
return;
--
2.34.3
[PATCH v14 4/4] vhost-vdpa: Add support for vIOMMU.
1. The vIOMMU support will make vDPA can work in IOMMU mode. This
will fix security issues while using the no-IOMMU mode.
To support this feature we need to add new functions for IOMMU MR adds and
deletes.
Also since the SVQ does not support vIOMMU yet, add the check for IOMMU
in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
the function will return fail.
2. Skip the iova_max check vhost_vdpa_listener_skipped_section(). While
MR is IOMMU, move this check to vhost_vdpa_iommu_map_notify()
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 149 +++--
include/hw/virtio/vhost-vdpa.h | 11 +++
2 files changed, 152 insertions(+), 8 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 0c8c37e786..b36922b365 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -60,15 +61,22 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, sometimes the section will be larger than iova_max,
+ * but the memory that actually maps is smaller, so move the check to
+ * function vhost_vdpa_iommu_map_notify(). That function will use the
actual
+ * size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
-
return false;
}
@@ -185,6 +193,118 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+Int128 llend;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+/* check if RAM section out of device range */
+llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
+if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ v->iova_range.last, int128_get64(llend));
+return;
+}
+
+vhost_vdpa_iotlb_batch_begin_once(v);
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+
+ret = vhost_vdpa_dma_map(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1, vaddr, read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+Int128 end;
+
[PATCH v14 0/4] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments fix the crash found in test changes in V12 Address the comments, squash patch 1 into the next patch improve the code style issue changes in V13 fail to start if IOMMU and svq enable at same time improve the code style issue changes in V14 Address the comments Cindy Lu (4): vhost: expose function vhost_dev_has_iommu() vhost_vdpa: fix the input in trace_vhost_vdpa_listener_region_del() vhost-vdpa: Add check for full 64-bit in region delete vhost-vdpa: Add support for vIOMMU. hw/virtio/vhost-vdpa.c | 172 +++-- hw/virtio/vhost.c | 2 +- include/hw/virtio/vhost-vdpa.h | 11 +++ include/hw/virtio/vhost.h | 1 + 4 files changed, 175 insertions(+), 11 deletions(-) -- 2.34.3
[PATCH v14 3/4] vhost-vdpa: Add check for full 64-bit in region delete
The unmap ioctl doesn't accept a full 64-bit span. So need to
add check for the section's size in vhost_vdpa_listener_region_del().
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 20 +++-
1 file changed, 19 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 92c2413c76..0c8c37e786 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -316,10 +316,28 @@ static void vhost_vdpa_listener_region_del(MemoryListener
*listener,
vhost_iova_tree_remove(v->iova_tree, *result);
}
vhost_vdpa_iotlb_batch_begin_once(v);
+/*
+ * The unmap ioctl doesn't accept a full 64-bit. need to check it
+ */
+if (int128_eq(llsize, int128_2_64())) {
+llsize = int128_rshift(llsize, 1);
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ int128_get64(llsize));
+
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
+}
+iova += int128_get64(llsize);
+}
ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
int128_get64(llsize));
+
if (ret) {
-error_report("vhost_vdpa dma unmap error!");
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, int128_get64(llsize), ret);
}
memory_region_unref(section->mr);
--
2.34.3
Re: [PATCH v13 2/2] vhost-vdpa: add support for vIOMMU
On Mon, Mar 6, 2023 at 11:36 AM Jason Wang wrote:
>
>
> 在 2023/2/8 10:57, Cindy Lu 写道:
> > 1.Add support for vIOMMU.
> > Add the new function to deal with iommu MR.
> > - during iommu_region_add register a specific IOMMU notifier,
> >and store all notifiers in a list.
> > - during iommu_region_del, compare and delete the IOMMU notifier from the
> > list
> > - since the SVQ not support iommu yet, add the check for IOMMU
> >in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
> >function will return fail.
> >
> > 2.Skip the check in vhost_vdpa_listener_skipped_section() while
> > MR is IOMMU, Move this check to vhost_vdpa_iommu_map_notify()
>
>
> This need some tweak as well, it's better not repeat what is done in the
> code but why do you need this change. More could be found at:
>
> https://docs.kernel.org/process/submitting-patches.html#describe-your-changes
>
sure, will change this
>
> >
> > Verified in vp_vdpa and vdpa_sim_net driver
> >
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/vhost-vdpa.c | 173 ++---
> > include/hw/virtio/vhost-vdpa.h | 11 +++
> > 2 files changed, 173 insertions(+), 11 deletions(-)
> >
> > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> > index 542e003101..46f676ab71 100644
> > --- a/hw/virtio/vhost-vdpa.c
> > +++ b/hw/virtio/vhost-vdpa.c
> > @@ -26,6 +26,7 @@
> > #include "cpu.h"
> > #include "trace.h"
> > #include "qapi/error.h"
> > +#include "hw/virtio/virtio-access.h"
> >
> > /*
> >* Return one past the end of the end of section. Be careful with uint64_t
> > @@ -60,15 +61,22 @@ static bool
> > vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
> >iova_min, section->offset_within_address_space);
> > return true;
> > }
> > +/*
> > + * While using vIOMMU, Sometimes the section will be larger than
> > iova_max
> > + * but the memory that actually mapping is smaller, So skip the check
> > + * here. Will add the check-in vhost_vdpa_iommu_map_notify,
> > + *There is the real size that maps to the kernel
>
>
> Please tweak the comment, it has issues of whitespace, capitalization,
> punctuation marks.
>
sure will change this
>
> > + */
> >
> > -llend = vhost_vdpa_section_end(section);
> > -if (int128_gt(llend, int128_make64(iova_max))) {
> > -error_report("RAM section out of device range (max=0x%" PRIx64
> > - ", end addr=0x%" PRIx64 ")",
> > - iova_max, int128_get64(llend));
> > -return true;
> > +if (!memory_region_is_iommu(section->mr)) {
>
>
> Note related to this patch but should we exclude non ram region here as
> well?
>
Sure, will add this check
>
> > +llend = vhost_vdpa_section_end(section);
> > +if (int128_gt(llend, int128_make64(iova_max))) {
> > +error_report("RAM section out of device range (max=0x%" PRIx64
> > + ", end addr=0x%" PRIx64 ")",
> > + iova_max, int128_get64(llend));
> > +return true;
> > +}
> > }
> > -
> > return false;
> > }
> >
> > @@ -185,6 +193,118 @@ static void vhost_vdpa_listener_commit(MemoryListener
> > *listener)
> > v->iotlb_batch_begin_sent = false;
> > }
> >
> > +static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry
> > *iotlb)
> > +{
> > +struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
> > +
> > +hwaddr iova = iotlb->iova + iommu->iommu_offset;
> > +struct vhost_vdpa *v = iommu->dev;
> > +void *vaddr;
> > +int ret;
> > +Int128 llend;
> > +
> > +if (iotlb->target_as != _space_memory) {
> > +error_report("Wrong target AS \"%s\", only system memory is
> > allowed",
> > + iotlb->target_as->name ? iotlb->target_as->name :
> > "none");
> > +return;
> > +}
> > +RCU_READ_LOCK_GUARD();
> > +/* check if RAM section out of device range */
> > +llend = int128_add(int128_makes64(iotlb->addr_mask),
> > int128_makes64(iova));
> > +if (int128_gt(llend, int128_make64(v->iova_range.las
[PATCH v13 2/2] vhost-vdpa: add support for vIOMMU
1.Add support for vIOMMU.
Add the new function to deal with iommu MR.
- during iommu_region_add register a specific IOMMU notifier,
and store all notifiers in a list.
- during iommu_region_del, compare and delete the IOMMU notifier from the list
- since the SVQ not support iommu yet, add the check for IOMMU
in vhost_vdpa_dev_start, if the SVQ and IOMMU enable at the same time
function will return fail.
2.Skip the check in vhost_vdpa_listener_skipped_section() while
MR is IOMMU, Move this check to vhost_vdpa_iommu_map_notify()
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 173 ++---
include/hw/virtio/vhost-vdpa.h | 11 +++
2 files changed, 173 insertions(+), 11 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 542e003101..46f676ab71 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -60,15 +61,22 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, Sometimes the section will be larger than iova_max
+ * but the memory that actually mapping is smaller, So skip the check
+ * here. Will add the check in vhost_vdpa_iommu_map_notify,
+ *There is the real size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
-
return false;
}
@@ -185,6 +193,118 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+Int128 llend;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+/* check if RAM section out of device range */
+llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
+if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ v->iova_range.last, int128_get64(llend));
+return;
+}
+
+vhost_vdpa_iotlb_batch_begin_once(v);
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+
+ret = vhost_vdpa_dma_map(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1, vaddr, read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, VHOST_VDPA_GUEST_PA_ASID, iova,
+ iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+
[PATCH v13 0/2] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments fix the crash found in test changes in V12 Address the comments, squash patch 1 into the next patch improve the code style issue changes in V13 fail to start if IOMMU and svq enable at same time improve the code style issue Cindy Lu (2): vhost: expose function vhost_dev_has_iommu() vhost-vdpa: add support for vIOMMU hw/virtio/vhost-vdpa.c | 173 ++--- hw/virtio/vhost.c | 2 +- include/hw/virtio/vhost-vdpa.h | 11 +++ include/hw/virtio/vhost.h | 1 + 4 files changed, 175 insertions(+), 12 deletions(-) -- 2.34.3
[PATCH v13 1/2] vhost: expose function vhost_dev_has_iommu()
To support vIOMMU in vdpa, need to exposed the function
vhost_dev_has_iommu, vdpa will use this function to check
if vIOMMU enable.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 2 +-
include/hw/virtio/vhost.h | 1 +
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index eb8c4c378c..9ff5516655 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -107,7 +107,7 @@ static void vhost_dev_sync_region(struct vhost_dev *dev,
}
}
-static bool vhost_dev_has_iommu(struct vhost_dev *dev)
+bool vhost_dev_has_iommu(struct vhost_dev *dev)
{
VirtIODevice *vdev = dev->vdev;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index a52f273347..f7f10c8fb7 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -336,4 +336,5 @@ int vhost_dev_set_inflight(struct vhost_dev *dev,
struct vhost_inflight *inflight);
int vhost_dev_get_inflight(struct vhost_dev *dev, uint16_t queue_size,
struct vhost_inflight *inflight);
+bool vhost_dev_has_iommu(struct vhost_dev *dev);
#endif
--
2.34.3
[PATCH v21 06/10] virtio: add support for configure interrupt
Add the functions to support the configure interrupt in virtio
The function virtio_config_guest_notifier_read will notify the
guest if there is an configure interrupt.
The function virtio_config_set_guest_notifier_fd_handler is
to set the fd hander for the notifier
Signed-off-by: Cindy Lu
---
hw/virtio/virtio.c | 29 +
include/hw/virtio/virtio.h | 4
2 files changed, 33 insertions(+)
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index eb6347ab5d..34e9c5d141 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -4012,7 +4012,14 @@ static void
virtio_queue_guest_notifier_read(EventNotifier *n)
virtio_irq(vq);
}
}
+static void virtio_config_guest_notifier_read(EventNotifier *n)
+{
+VirtIODevice *vdev = container_of(n, VirtIODevice, config_notifier);
+if (event_notifier_test_and_clear(n)) {
+virtio_notify_config(vdev);
+}
+}
void virtio_queue_set_guest_notifier_fd_handler(VirtQueue *vq, bool assign,
bool with_irqfd)
{
@@ -4029,6 +4036,23 @@ void
virtio_queue_set_guest_notifier_fd_handler(VirtQueue *vq, bool assign,
}
}
+void virtio_config_set_guest_notifier_fd_handler(VirtIODevice *vdev,
+ bool assign, bool with_irqfd)
+{
+EventNotifier *n;
+n = >config_notifier;
+if (assign && !with_irqfd) {
+event_notifier_set_handler(n, virtio_config_guest_notifier_read);
+} else {
+event_notifier_set_handler(n, NULL);
+}
+if (!assign) {
+/* Test and clear notifier before closing it,*/
+/* in case poll callback didn't have time to run. */
+virtio_config_guest_notifier_read(n);
+}
+}
+
EventNotifier *virtio_queue_get_guest_notifier(VirtQueue *vq)
{
return >guest_notifier;
@@ -4109,6 +4133,11 @@ EventNotifier *virtio_queue_get_host_notifier(VirtQueue
*vq)
return >host_notifier;
}
+EventNotifier *virtio_config_get_guest_notifier(VirtIODevice *vdev)
+{
+return >config_notifier;
+}
+
void virtio_queue_set_host_notifier_enabled(VirtQueue *vq, bool enabled)
{
vq->host_notifier_enabled = enabled;
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 1f4a41b958..9c3a4642f2 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -138,6 +138,7 @@ struct VirtIODevice
AddressSpace *dma_as;
QLIST_HEAD(, VirtQueue) *vector_queues;
QTAILQ_ENTRY(VirtIODevice) next;
+EventNotifier config_notifier;
};
struct VirtioDeviceClass {
@@ -360,6 +361,9 @@ void
virtio_queue_aio_attach_host_notifier_no_poll(VirtQueue *vq, AioContext *ct
void virtio_queue_aio_detach_host_notifier(VirtQueue *vq, AioContext *ctx);
VirtQueue *virtio_vector_first_queue(VirtIODevice *vdev, uint16_t vector);
VirtQueue *virtio_vector_next_queue(VirtQueue *vq);
+EventNotifier *virtio_config_get_guest_notifier(VirtIODevice *vdev);
+void virtio_config_set_guest_notifier_fd_handler(VirtIODevice *vdev,
+ bool assign, bool with_irqfd);
static inline void virtio_add_feature(uint64_t *features, unsigned int fbit)
{
--
2.34.3
[PATCH v21 07/10] vhost: add support for configure interrupt
Add functions to support configure interrupt.
The configure interrupt process will start in vhost_dev_start
and stop in vhost_dev_stop.
Also add the functions to support vhost_config_pending and
vhost_config_mask.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 78 ++-
include/hw/virtio/vhost.h | 4 ++
2 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 7fb008bc9e..84dbb39e07 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1596,7 +1596,68 @@ void vhost_virtqueue_mask(struct vhost_dev *hdev,
VirtIODevice *vdev, int n,
file.index = hdev->vhost_ops->vhost_get_vq_index(hdev, n);
r = hdev->vhost_ops->vhost_set_vring_call(hdev, );
if (r < 0) {
-VHOST_OPS_DEBUG(r, "vhost_set_vring_call failed");
+error_report("vhost_set_vring_call failed %d", -r);
+}
+}
+
+bool vhost_config_pending(struct vhost_dev *hdev)
+{
+assert(hdev->vhost_ops);
+if ((hdev->started == false) ||
+(hdev->vhost_ops->vhost_set_config_call == NULL)) {
+return false;
+}
+
+EventNotifier *notifier =
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier;
+return event_notifier_test_and_clear(notifier);
+}
+
+void vhost_config_mask(struct vhost_dev *hdev, VirtIODevice *vdev, bool mask)
+{
+int fd;
+int r;
+EventNotifier *notifier =
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier;
+EventNotifier *config_notifier = >config_notifier;
+assert(hdev->vhost_ops);
+
+if ((hdev->started == false) ||
+(hdev->vhost_ops->vhost_set_config_call == NULL)) {
+return;
+}
+if (mask) {
+assert(vdev->use_guest_notifier_mask);
+fd = event_notifier_get_fd(notifier);
+} else {
+fd = event_notifier_get_fd(config_notifier);
+}
+r = hdev->vhost_ops->vhost_set_config_call(hdev, fd);
+if (r < 0) {
+error_report("vhost_set_config_call failed %d", -r);
+}
+}
+
+static void vhost_stop_config_intr(struct vhost_dev *dev)
+{
+int fd = -1;
+assert(dev->vhost_ops);
+if (dev->vhost_ops->vhost_set_config_call) {
+dev->vhost_ops->vhost_set_config_call(dev, fd);
+}
+}
+
+static void vhost_start_config_intr(struct vhost_dev *dev)
+{
+int r;
+
+assert(dev->vhost_ops);
+int fd = event_notifier_get_fd(>vdev->config_notifier);
+if (dev->vhost_ops->vhost_set_config_call) {
+r = dev->vhost_ops->vhost_set_config_call(dev, fd);
+if (!r) {
+event_notifier_set(>vdev->config_notifier);
+}
}
}
@@ -1836,6 +1897,16 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
}
}
+r = event_notifier_init(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier, 0);
+if (r < 0) {
+return r;
+}
+event_notifier_test_and_clear(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier);
+if (!vdev->use_guest_notifier_mask) {
+vhost_config_mask(hdev, vdev, true);
+}
if (hdev->log_enabled) {
uint64_t log_base;
@@ -1874,6 +1945,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
vhost_device_iotlb_miss(hdev, vq->used_phys, true);
}
}
+vhost_start_config_intr(hdev);
return 0;
fail_start:
if (vrings) {
@@ -1903,6 +1975,9 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
/* should only be called after backend is connected */
assert(hdev->vhost_ops);
+event_notifier_test_and_clear(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier);
+event_notifier_test_and_clear(>config_notifier);
trace_vhost_dev_stop(hdev, vdev->name, vrings);
@@ -1925,6 +2000,7 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
}
memory_listener_unregister(>iommu_listener);
}
+vhost_stop_config_intr(hdev);
vhost_log_put(hdev, true);
hdev->started = false;
vdev->vhost_started = false;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 67a6807fac..05bedb2416 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -33,6 +33,7 @@ struct vhost_virtqueue {
unsigned used_size;
EventNotifier masked_notifier;
EventNotifier error_notifier;
+EventNotifier masked_config_notifier;
struct vhost_dev *dev;
};
@@ -41,6 +42,7 @@ typedef unsigned long vhost_log_chunk_t;
#define VHOST_LOG_BITS (8 * sizeof(vhost_log_chunk_t))
#define VHOST_LOG_CHUNK (VHOST_LOG_PAGE * VHOST_LOG_BITS)
#define VHOST_INVALID_FEATURE_BIT (0xff)
+#define VHOST_QUEUE_NUM_CONFIG_INR 0
struct vhost_log {
unsigned lo
[PATCH v21 08/10] virtio-net: add support for configure interrupt
Add functions to support configure interrupt in virtio_net
Add the functions to support vhost_net_config_pending
and vhost_net_config_mask.
Signed-off-by: Cindy Lu
---
hw/net/vhost_net-stub.c | 9 +
hw/net/vhost_net.c | 9 +
hw/net/virtio-net.c | 4 ++--
include/net/vhost_net.h | 2 ++
4 files changed, 22 insertions(+), 2 deletions(-)
diff --git a/hw/net/vhost_net-stub.c b/hw/net/vhost_net-stub.c
index 9f7daae99c..c36f258201 100644
--- a/hw/net/vhost_net-stub.c
+++ b/hw/net/vhost_net-stub.c
@@ -82,6 +82,15 @@ void vhost_net_virtqueue_mask(VHostNetState *net,
VirtIODevice *dev,
{
}
+bool vhost_net_config_pending(VHostNetState *net)
+{
+return false;
+}
+
+void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool mask)
+{
+}
+
int vhost_net_notify_migration_done(struct vhost_net *net, char* mac_addr)
{
return -1;
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 043058ff43..6a55f5a473 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -478,6 +478,15 @@ void vhost_net_virtqueue_mask(VHostNetState *net,
VirtIODevice *dev,
vhost_virtqueue_mask(>dev, dev, idx, mask);
}
+bool vhost_net_config_pending(VHostNetState *net)
+{
+return vhost_config_pending(>dev);
+}
+
+void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool mask)
+{
+vhost_config_mask(>dev, dev, mask);
+}
VHostNetState *get_vhost_net(NetClientState *nc)
{
VHostNetState *vhost_net = 0;
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index bee35d6f9f..ec974f7a76 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3323,7 +3323,7 @@ static bool
virtio_net_guest_notifier_pending(VirtIODevice *vdev, int idx)
*/
if (idx == VIRTIO_CONFIG_IRQ_IDX) {
-return false;
+return vhost_net_config_pending(get_vhost_net(nc->peer));
}
return vhost_net_virtqueue_pending(get_vhost_net(nc->peer), idx);
}
@@ -3355,9 +3355,9 @@ static void virtio_net_guest_notifier_mask(VirtIODevice
*vdev, int idx,
*/
if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+vhost_net_config_mask(get_vhost_net(nc->peer), vdev, mask);
return;
}
-
vhost_net_virtqueue_mask(get_vhost_net(nc->peer), vdev, idx, mask);
}
diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h
index 40b9a40074..dbbd0dc04e 100644
--- a/include/net/vhost_net.h
+++ b/include/net/vhost_net.h
@@ -39,6 +39,8 @@ int vhost_net_set_config(struct vhost_net *net, const uint8_t
*data,
bool vhost_net_virtqueue_pending(VHostNetState *net, int n);
void vhost_net_virtqueue_mask(VHostNetState *net, VirtIODevice *dev,
int idx, bool mask);
+bool vhost_net_config_pending(VHostNetState *net);
+void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool mask);
int vhost_net_notify_migration_done(VHostNetState *net, char* mac_addr);
VHostNetState *get_vhost_net(NetClientState *nc);
--
2.34.3
[PATCH v21 09/10] virtio-mmio: add support for configure interrupt
Add configure interrupt support in virtio-mmio bus.
add function to set configure guest notifier.
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-mmio.c | 27 +++
1 file changed, 27 insertions(+)
diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index d240efef97..103260ec15 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -670,7 +670,30 @@ static int virtio_mmio_set_guest_notifier(DeviceState *d,
int n, bool assign,
return 0;
}
+static int virtio_mmio_set_config_guest_notifier(DeviceState *d, bool assign,
+ bool with_irqfd)
+{
+VirtIOMMIOProxy *proxy = VIRTIO_MMIO(d);
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
+EventNotifier *notifier = virtio_config_get_guest_notifier(vdev);
+int r = 0;
+if (assign) {
+r = event_notifier_init(notifier, 0);
+if (r < 0) {
+return r;
+}
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+} else {
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+event_notifier_cleanup(notifier);
+}
+if (vdc->guest_notifier_mask && vdev->use_guest_notifier_mask) {
+vdc->guest_notifier_mask(vdev, VIRTIO_CONFIG_IRQ_IDX, !assign);
+}
+return r;
+}
static int virtio_mmio_set_guest_notifiers(DeviceState *d, int nvqs,
bool assign)
{
@@ -692,6 +715,10 @@ static int virtio_mmio_set_guest_notifiers(DeviceState *d,
int nvqs,
goto assign_error;
}
}
+r = virtio_mmio_set_config_guest_notifier(d, assign, with_irqfd);
+if (r < 0) {
+goto assign_error;
+}
return 0;
--
2.34.3
[PATCH v21 10/10] virtio-pci: add support for configure interrupt
Add process to handle the configure interrupt, The function's
logic is the same with vq interrupt.Add extra process to check
the configure interrupt
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 118 +++--
include/hw/virtio/virtio-pci.h | 4 +-
2 files changed, 102 insertions(+), 20 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index ec816ea367..3f00e91718 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -751,7 +751,8 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy *proxy,
int queue_no,
VirtQueue *vq;
if (queue_no == VIRTIO_CONFIG_IRQ_IDX) {
-return -1;
+*n = virtio_config_get_guest_notifier(vdev);
+*vector = vdev->config_vector;
} else {
if (!virtio_queue_get_num(vdev, queue_no)) {
return -1;
@@ -811,7 +812,7 @@ undo:
}
return ret;
}
-static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+static int kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy *proxy, int nvqs)
{
int queue_no;
int ret = 0;
@@ -826,6 +827,10 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy
*proxy, int nvqs)
return ret;
}
+static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
+{
+return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
+}
static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
int queue_no)
@@ -850,7 +855,7 @@ static void
kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
kvm_virtio_pci_vq_vector_release(proxy, vector);
}
-static void kvm_virtio_pci_vector_release(VirtIOPCIProxy *proxy, int nvqs)
+static void kvm_virtio_pci_vector_vq_release(VirtIOPCIProxy *proxy, int nvqs)
{
int queue_no;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
@@ -863,6 +868,11 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
}
}
+static void kvm_virtio_pci_vector_config_release(VirtIOPCIProxy *proxy)
+{
+kvm_virtio_pci_vector_release_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
+}
+
static int virtio_pci_one_vector_unmask(VirtIOPCIProxy *proxy,
unsigned int queue_no,
unsigned int vector,
@@ -944,9 +954,19 @@ static int virtio_pci_vector_unmask(PCIDevice *dev,
unsigned vector,
}
vq = virtio_vector_next_queue(vq);
}
-
+/* unmask config intr */
+if (vector == vdev->config_vector) {
+n = virtio_config_get_guest_notifier(vdev);
+ret = virtio_pci_one_vector_unmask(proxy, VIRTIO_CONFIG_IRQ_IDX,
vector,
+ msg, n);
+if (ret < 0) {
+goto undo_config;
+}
+}
return 0;
-
+undo_config:
+n = virtio_config_get_guest_notifier(vdev);
+virtio_pci_one_vector_mask(proxy, VIRTIO_CONFIG_IRQ_IDX, vector, n);
undo:
vq = virtio_vector_first_queue(vdev, vector);
while (vq && unmasked >= 0) {
@@ -980,6 +1000,11 @@ static void virtio_pci_vector_mask(PCIDevice *dev,
unsigned vector)
}
vq = virtio_vector_next_queue(vq);
}
+
+if (vector == vdev->config_vector) {
+n = virtio_config_get_guest_notifier(vdev);
+virtio_pci_one_vector_mask(proxy, VIRTIO_CONFIG_IRQ_IDX, vector, n);
+}
}
static void virtio_pci_vector_poll(PCIDevice *dev,
@@ -1011,6 +1036,34 @@ static void virtio_pci_vector_poll(PCIDevice *dev,
msix_set_pending(dev, vector);
}
}
+/* poll the config intr */
+ret = virtio_pci_get_notifier(proxy, VIRTIO_CONFIG_IRQ_IDX, ,
+ );
+if (ret < 0) {
+return;
+}
+if (vector < vector_start || vector >= vector_end ||
+!msix_is_masked(dev, vector)) {
+return;
+}
+if (k->guest_notifier_pending) {
+if (k->guest_notifier_pending(vdev, VIRTIO_CONFIG_IRQ_IDX)) {
+msix_set_pending(dev, vector);
+}
+} else if (event_notifier_test_and_clear(notifier)) {
+msix_set_pending(dev, vector);
+}
+}
+
+void virtio_pci_set_guest_notifier_fd_handler(VirtIODevice *vdev, VirtQueue
*vq,
+ int n, bool assign,
+ bool with_irqfd)
+{
+if (n == VIRTIO_CONFIG_IRQ_IDX) {
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+} else {
+virtio_queue_set_guest_notifier_fd_handler(vq, assign, with_irqfd);
+}
}
static int virtio_pci_set_guest_notifier(DeviceState *d, int n, bool assign,
@@ -1019,17 +1072,25 @@ static int virtio_pci_set_guest_notifier(DeviceState
*d, int n, bool assign,
VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(
[PATCH v21 05/10] vhost-vdpa: add support for config interrupt
Add new call back function in vhost-vdpa, The function
vhost_set_config_call can set the event fd to kernel.
This function will be called in the vhost_dev_start
and vhost_dev_stop
Signed-off-by: Cindy Lu
---
hw/virtio/trace-events | 1 +
hw/virtio/vhost-vdpa.c | 8
2 files changed, 9 insertions(+)
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index 14fc5b9bb2..46f2faf04e 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -62,6 +62,7 @@ vhost_vdpa_get_features(void *dev, uint64_t features) "dev:
%p features: 0x%"PRI
vhost_vdpa_set_owner(void *dev) "dev: %p"
vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t
avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr:
0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64
vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p
first: 0x%"PRIx64" last: 0x%"PRIx64
+vhost_vdpa_set_config_call(void *dev, int fd)"dev: %p fd: %d"
# virtio.c
virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned
out_num) "elem %p size %zd in_num %u out_num %u"
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 7468e44b87..c5be2645b0 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -754,6 +754,13 @@ static int vhost_vdpa_set_vring_ready(struct vhost_dev
*dev)
return 0;
}
+static int vhost_vdpa_set_config_call(struct vhost_dev *dev,
+ int fd)
+{
+trace_vhost_vdpa_set_config_call(dev, fd);
+return vhost_vdpa_call(dev, VHOST_VDPA_SET_CONFIG_CALL, );
+}
+
static void vhost_vdpa_dump_config(struct vhost_dev *dev, const uint8_t
*config,
uint32_t config_len)
{
@@ -1310,4 +1317,5 @@ const VhostOps vdpa_ops = {
.vhost_get_device_id = vhost_vdpa_get_device_id,
.vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
.vhost_force_iommu = vhost_vdpa_force_iommu,
+.vhost_set_config_call = vhost_vdpa_set_config_call,
};
--
2.34.3
[PATCH v21 03/10] virtio-pci: decouple the single vector from the interrupt process
To reuse the interrupt process in configure interrupt
Need to decouple the single vector from the interrupt process.
We add new function kvm_virtio_pci_vector_use_one and _release_one.
These functions are used for the single vector, the whole process will
finish in the loop with vq number.
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 131 +++--
1 file changed, 73 insertions(+), 58 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 52c7692fff..ec816ea367 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -699,7 +699,6 @@ static uint32_t virtio_read_config(PCIDevice *pci_dev,
}
static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,
-unsigned int queue_no,
unsigned int vector)
{
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
@@ -764,87 +763,103 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy
*proxy, int queue_no,
return 0;
}
-static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
{
+unsigned int vector;
+int ret;
+EventNotifier *n;
PCIDevice *dev = >pci_dev;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
-unsigned int vector;
-int ret, queue_no;
-EventNotifier *n;
-for (queue_no = 0; queue_no < nvqs; queue_no++) {
-if (!virtio_queue_get_num(vdev, queue_no)) {
-break;
-}
-ret = virtio_pci_get_notifier(proxy, queue_no, , );
-if (ret < 0) {
-break;
-}
-if (vector >= msix_nr_vectors_allocated(dev)) {
-continue;
-}
-ret = kvm_virtio_pci_vq_vector_use(proxy, queue_no, vector);
+
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+return ret;
+}
+if (vector >= msix_nr_vectors_allocated(dev)) {
+return 0;
+}
+ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
+if (ret < 0) {
+goto undo;
+}
+/*
+ * If guest supports masking, set up irqfd now.
+ * Otherwise, delay until unmasked in the frontend.
+ */
+if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
+ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
if (ret < 0) {
+kvm_virtio_pci_vq_vector_release(proxy, vector);
goto undo;
}
-/* If guest supports masking, set up irqfd now.
- * Otherwise, delay until unmasked in the frontend.
- */
-if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
-if (ret < 0) {
-kvm_virtio_pci_vq_vector_release(proxy, vector);
-goto undo;
-}
-}
}
-return 0;
+return 0;
undo:
-while (--queue_no >= 0) {
-vector = virtio_queue_vector(vdev, queue_no);
-if (vector >= msix_nr_vectors_allocated(dev)) {
-continue;
+
+vector = virtio_queue_vector(vdev, queue_no);
+if (vector >= msix_nr_vectors_allocated(dev)) {
+return ret;
+}
+if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+return ret;
}
-if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = virtio_pci_get_notifier(proxy, queue_no, , );
-if (ret < 0) {
-break;
-}
-kvm_virtio_pci_irqfd_release(proxy, n, vector);
+kvm_virtio_pci_irqfd_release(proxy, n, vector);
+}
+return ret;
+}
+static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+{
+int queue_no;
+int ret = 0;
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+
+for (queue_no = 0; queue_no < nvqs; queue_no++) {
+if (!virtio_queue_get_num(vdev, queue_no)) {
+return -1;
}
-kvm_virtio_pci_vq_vector_release(proxy, vector);
+ret = kvm_virtio_pci_vector_use_one(proxy, queue_no);
}
return ret;
}
-static void kvm_virtio_pci_vector_release(VirtIOPCIProxy *proxy, int nvqs)
+
+static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
+ int queue_no)
{
-PCIDevice *dev = >pci_dev;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
unsigned int vector;
-int queue_no;
-VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
EventNotifier *n;
-int ret ;
+int ret;
+VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+PCIDevice *dev = >pci_dev;
[PATCH v21 01/10] virtio: introduce macro VIRTIO_CONFIG_IRQ_IDX
To support configure interrupt for vhost-vdpa
Introduce VIRTIO_CONFIG_IRQ_IDX -1 as configure interrupt's queue index,
Then we can reuse the functions guest_notifier_mask and guest_notifier_pending.
Add the check of queue index in these drivers, if the driver does not support
configure interrupt, the function will just return
Signed-off-by: Cindy Lu
---
hw/display/vhost-user-gpu.c| 18 ++
hw/net/virtio-net.c| 22 --
hw/virtio/vhost-user-fs.c | 18 ++
hw/virtio/vhost-user-gpio.c| 10 ++
hw/virtio/vhost-vsock-common.c | 18 ++
hw/virtio/virtio-crypto.c | 18 ++
include/hw/virtio/virtio.h | 3 +++
7 files changed, 105 insertions(+), 2 deletions(-)
diff --git a/hw/display/vhost-user-gpu.c b/hw/display/vhost-user-gpu.c
index 19c0e20103..4380a5e672 100644
--- a/hw/display/vhost-user-gpu.c
+++ b/hw/display/vhost-user-gpu.c
@@ -486,6 +486,15 @@ vhost_user_gpu_guest_notifier_pending(VirtIODevice *vdev,
int idx)
{
VhostUserGPU *g = VHOST_USER_GPU(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_virtqueue_pending(>vhost->dev, idx);
}
@@ -494,6 +503,15 @@ vhost_user_gpu_guest_notifier_mask(VirtIODevice *vdev, int
idx, bool mask)
{
VhostUserGPU *g = VHOST_USER_GPU(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
vhost_virtqueue_mask(>vhost->dev, vdev, idx, mask);
}
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index aba12759d5..bee35d6f9f 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3316,6 +3316,15 @@ static bool
virtio_net_guest_notifier_pending(VirtIODevice *vdev, int idx)
} else {
nc = qemu_get_subqueue(n->nic, vq2q(idx));
}
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return false
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_net_virtqueue_pending(get_vhost_net(nc->peer), idx);
}
@@ -3339,8 +3348,17 @@ static void virtio_net_guest_notifier_mask(VirtIODevice
*vdev, int idx,
} else {
nc = qemu_get_subqueue(n->nic, vq2q(idx));
}
-vhost_net_virtqueue_mask(get_vhost_net(nc->peer),
- vdev, idx, mask);
+/*
+ *Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
+
+vhost_net_virtqueue_mask(get_vhost_net(nc->peer), vdev, idx, mask);
}
static void virtio_net_set_config_size(VirtIONet *n, uint64_t host_features)
diff --git a/hw/virtio/vhost-user-fs.c b/hw/virtio/vhost-user-fs.c
index d97b179e6f..f5049735ac 100644
--- a/hw/virtio/vhost-user-fs.c
+++ b/hw/virtio/vhost-user-fs.c
@@ -159,6 +159,15 @@ static void vuf_guest_notifier_mask(VirtIODevice *vdev,
int idx,
{
VHostUserFS *fs = VHOST_USER_FS(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
vhost_virtqueue_mask(>vhost_dev, vdev, idx, mask);
}
@@ -166,6 +175,15 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev,
int idx)
{
VHostUserFS *fs = VHOST_USER_FS(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_virtqueue_pending(>vhost_dev, idx);
}
diff --git a/hw/virtio/vhost-user-gpio.c b/hw/virtio/vhost-user-gpio.c
index b7b82a1099..fe3da32c74 100644
--- a/hw/virtio/vhost-user-gpio.c
+++ b/hw/virtio/vhost-user-gpio.c
@@ -191,6 +191,16 @@ static void vu_gpio_guest_notifier_mask(VirtIODevice
*vdev, int idx, bool mask)
{
VHostUserGPIO *gpio = VHOST_USER_GPIO(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function wi
[PATCH v21 00/10] vhost-vdpa: add support for configure interrupt
These patches introduced the support for configure interrupt These codes are tested on x86_64 and aarch64 platforms. the test scenario is based on vp-vdpa/vdpa_sim_net /vhost/vhost_user/testpmd, with/without irqfd. Tested in virtio-pci bus and virtio-mmio bus Change in v2: Add support for virtio-mmio bus active the notifier while the backend support configure interrupt misc fixes from v1 Change in v3 fix the coding style problems Change in v4 misc fixes from v3 merge the set_config_notifier to set_guest_notifier when vdpa start, check the feature by VIRTIO_NET_F_STATUS Change in v5 misc fixes from v4 split the code to introduce configure interrupt type and the callback function will init the configure interrupt in all virtio-pci and virtio-mmio bus, but will only active while using vhost-vdpa driver Change in v6 misc fixes from v5 decouple vq from interrupt setting and misc process fix the bug in virtio_net_handle_rx Change in v7 misc fixes from v6 decouple vq from interrupt setting and misc process decouple vq from vector use/release process decouple vq from set notifier fd handler process move config_notifier and masked_config_notifier to VirtIODevice fix the bug in virtio_net_handle_rx, add more information add VIRTIO_CONFIG_IRQ_IDX as the queue number for configure interrupt Change in v8 misc fixes from v7 decouple vq from interrupt setting and misc process decouple vq from vector use/release process decouple vq from set notifier fd handler process move the vhost configure interrupt to vhost_net Change in v9 misc fixes from v8 address the comments from v8 Change in v10 fix the hang issue in qtest address the comments from v9 Change in v11 fix the crash in aarch64 plateform. fix the crash upstream reported Change in v12 fix the typo and the comments changes in v13 re-send the patches by git-publish changes in v14 rebased the code based on upstream changes in v15 rebased the code based on upstream changes in v16 resend the patch, seems there not send successfully changes in v18 fix the crash in qtest Because hw/virtio/vhost-user-gpio.c is a new device,So I missed to add the VIRTIO_CONFIG_IRQ_IDX check in notifier_mask changes in v19 fix the crash in qtest rebased the code based on upstream changes in v20 fix the compile issue in mingw32 Cindy Lu (10): virtio: introduce macro VIRTIO_CONFIG_IRQ_IDX virtio-pci: decouple notifier from interrupt process virtio-pci: decouple the single vector from the interrupt process vhost: introduce new VhostOps vhost_set_config_call vhost-vdpa: add support for config interrupt virtio: add support for configure interrupt vhost: add support for configure interrupt virtio-net: add support for configure interrupt virtio-mmio: add support for configure interrupt virtio-pci: add support for configure interrupt hw/display/vhost-user-gpu.c | 18 ++ hw/net/vhost_net-stub.c | 9 + hw/net/vhost_net.c| 9 + hw/net/virtio-net.c | 22 ++- hw/virtio/trace-events| 1 + hw/virtio/vhost-user-fs.c | 18 ++ hw/virtio/vhost-user-gpio.c | 10 + hw/virtio/vhost-vdpa.c| 8 + hw/virtio/vhost-vsock-common.c| 18 ++ hw/virtio/vhost.c | 78 +++- hw/virtio/virtio-crypto.c | 18 ++ hw/virtio/virtio-mmio.c | 27 +++ hw/virtio/virtio-pci.c| 291 +- hw/virtio/virtio.c| 29 +++ include/hw/virtio/vhost-backend.h | 3 + include/hw/virtio/vhost.h | 4 + include/hw/virtio/virtio-pci.h| 4 +- include/hw/virtio/virtio.h| 7 + include/net/vhost_net.h | 2 + 19 files changed, 487 insertions(+), 89 deletions(-) -- 2.34.3
[PATCH v21 04/10] vhost: introduce new VhostOps vhost_set_config_call
This patch introduces new VhostOps vhost_set_config_call.
This function allows the qemu to set the config
event fd to kernel driver.
Signed-off-by: Cindy Lu
---
include/hw/virtio/vhost-backend.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/include/hw/virtio/vhost-backend.h
b/include/hw/virtio/vhost-backend.h
index eab46d7f0b..c5ab49051e 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -128,6 +128,8 @@ typedef int (*vhost_get_device_id_op)(struct vhost_dev
*dev, uint32_t *dev_id);
typedef bool (*vhost_force_iommu_op)(struct vhost_dev *dev);
+typedef int (*vhost_set_config_call_op)(struct vhost_dev *dev,
+ int fd);
typedef struct VhostOps {
VhostBackendType backend_type;
vhost_backend_init vhost_backend_init;
@@ -174,6 +176,7 @@ typedef struct VhostOps {
vhost_vq_get_addr_op vhost_vq_get_addr;
vhost_get_device_id_op vhost_get_device_id;
vhost_force_iommu_op vhost_force_iommu;
+vhost_set_config_call_op vhost_set_config_call;
} VhostOps;
int vhost_backend_update_device_iotlb(struct vhost_dev *dev,
--
2.34.3
[PATCH v21 02/10] virtio-pci: decouple notifier from interrupt process
To reuse the notifier process. We add the virtio_pci_get_notifier
to get the notifier and vector. The INPUT for this function is IDX,
The OUTPUT is the notifier and the vector
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 88 +++---
1 file changed, 57 insertions(+), 31 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index a1c9dfa7bb..52c7692fff 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -728,29 +728,41 @@ static void
kvm_virtio_pci_vq_vector_release(VirtIOPCIProxy *proxy,
}
static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,
- unsigned int queue_no,
+ EventNotifier *n,
unsigned int vector)
{
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
-VirtIODevice *vdev = virtio_bus_get_device(>bus);
-VirtQueue *vq = virtio_get_queue(vdev, queue_no);
-EventNotifier *n = virtio_queue_get_guest_notifier(vq);
return kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, n, NULL, irqfd->virq);
}
static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,
- unsigned int queue_no,
+ EventNotifier *n ,
unsigned int vector)
{
-VirtIODevice *vdev = virtio_bus_get_device(>bus);
-VirtQueue *vq = virtio_get_queue(vdev, queue_no);
-EventNotifier *n = virtio_queue_get_guest_notifier(vq);
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
int ret;
ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, n, irqfd->virq);
assert(ret == 0);
}
+static int virtio_pci_get_notifier(VirtIOPCIProxy *proxy, int queue_no,
+ EventNotifier **n, unsigned int *vector)
+{
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+VirtQueue *vq;
+
+if (queue_no == VIRTIO_CONFIG_IRQ_IDX) {
+return -1;
+} else {
+if (!virtio_queue_get_num(vdev, queue_no)) {
+return -1;
+}
+*vector = virtio_queue_vector(vdev, queue_no);
+vq = virtio_get_queue(vdev, queue_no);
+*n = virtio_queue_get_guest_notifier(vq);
+}
+return 0;
+}
static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
{
@@ -759,12 +771,15 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy
*proxy, int nvqs)
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
unsigned int vector;
int ret, queue_no;
-
+EventNotifier *n;
for (queue_no = 0; queue_no < nvqs; queue_no++) {
if (!virtio_queue_get_num(vdev, queue_no)) {
break;
}
-vector = virtio_queue_vector(vdev, queue_no);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
if (vector >= msix_nr_vectors_allocated(dev)) {
continue;
}
@@ -776,7 +791,7 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy,
int nvqs)
* Otherwise, delay until unmasked in the frontend.
*/
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = kvm_virtio_pci_irqfd_use(proxy, queue_no, vector);
+ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
if (ret < 0) {
kvm_virtio_pci_vq_vector_release(proxy, vector);
goto undo;
@@ -792,7 +807,11 @@ undo:
continue;
}
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-kvm_virtio_pci_irqfd_release(proxy, queue_no, vector);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
+kvm_virtio_pci_irqfd_release(proxy, n, vector);
}
kvm_virtio_pci_vq_vector_release(proxy, vector);
}
@@ -806,12 +825,16 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
unsigned int vector;
int queue_no;
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
-
+EventNotifier *n;
+int ret ;
for (queue_no = 0; queue_no < nvqs; queue_no++) {
if (!virtio_queue_get_num(vdev, queue_no)) {
break;
}
-vector = virtio_queue_vector(vdev, queue_no);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
if (vector >= msix_nr_vectors_allocated(dev)) {
continue;
}
@@ -819,21 +842,20 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
* Otherwise, it was cleaned when masked in the frontend.
*/
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-kvm_virtio_pci_irqfd_relea
Re: [PATCH v20 08/10] virtio-net: add support for configure interrupt
On Wed, 21 Dec 2022 at 19:25, Michael S. Tsirkin wrote:
>
> On Mon, Dec 12, 2022 at 01:20:40AM +0800, Cindy Lu wrote:
> > Add functions to support configure interrupt in virtio_net
> > Add the functions to support vhost_net_config_pending
> > and vhost_net_config_mask.
> >
> > Signed-off-by: Cindy Lu
>
>
>
> Causes a bunch of failures including mingw.
> See:
> https://gitlab.com/mstredhat/qemu/-/pipelines/728492799/failures
>
> lots of:
> /builds/mstredhat/qemu/build/../hw/net/virtio-net.c:3366: undefined reference
> to `vhost_net_config_mask'
>
> Pls fix and repost.
>
Sorry, Michael, will post a new version soon
>
> > ---
> > hw/net/vhost_net.c | 9 +
> > hw/net/virtio-net.c | 4 ++--
> > include/net/vhost_net.h | 2 ++
> > 3 files changed, 13 insertions(+), 2 deletions(-)
> >
> > diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
> > index 043058ff43..6a55f5a473 100644
> > --- a/hw/net/vhost_net.c
> > +++ b/hw/net/vhost_net.c
> > @@ -478,6 +478,15 @@ void vhost_net_virtqueue_mask(VHostNetState *net,
> > VirtIODevice *dev,
> > vhost_virtqueue_mask(>dev, dev, idx, mask);
> > }
> >
> > +bool vhost_net_config_pending(VHostNetState *net)
> > +{
> > +return vhost_config_pending(>dev);
> > +}
> > +
> > +void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool
> > mask)
> > +{
> > +vhost_config_mask(>dev, dev, mask);
> > +}
> > VHostNetState *get_vhost_net(NetClientState *nc)
> > {
> > VHostNetState *vhost_net = 0;
> > diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
> > index bee35d6f9f..ec974f7a76 100644
> > --- a/hw/net/virtio-net.c
> > +++ b/hw/net/virtio-net.c
> > @@ -3323,7 +3323,7 @@ static bool
> > virtio_net_guest_notifier_pending(VirtIODevice *vdev, int idx)
> > */
> >
> > if (idx == VIRTIO_CONFIG_IRQ_IDX) {
> > -return false;
> > +return vhost_net_config_pending(get_vhost_net(nc->peer));
> > }
> > return vhost_net_virtqueue_pending(get_vhost_net(nc->peer), idx);
> > }
> > @@ -3355,9 +3355,9 @@ static void
> > virtio_net_guest_notifier_mask(VirtIODevice *vdev, int idx,
> > */
> >
> > if (idx == VIRTIO_CONFIG_IRQ_IDX) {
> > +vhost_net_config_mask(get_vhost_net(nc->peer), vdev, mask);
> > return;
> > }
> > -
> > vhost_net_virtqueue_mask(get_vhost_net(nc->peer), vdev, idx, mask);
> > }
> >
> > diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h
> > index 40b9a40074..dbbd0dc04e 100644
> > --- a/include/net/vhost_net.h
> > +++ b/include/net/vhost_net.h
> > @@ -39,6 +39,8 @@ int vhost_net_set_config(struct vhost_net *net, const
> > uint8_t *data,
> > bool vhost_net_virtqueue_pending(VHostNetState *net, int n);
> > void vhost_net_virtqueue_mask(VHostNetState *net, VirtIODevice *dev,
> >int idx, bool mask);
> > +bool vhost_net_config_pending(VHostNetState *net);
> > +void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool
> > mask);
> > int vhost_net_notify_migration_done(VHostNetState *net, char* mac_addr);
> > VHostNetState *get_vhost_net(NetClientState *nc);
> >
> > --
> > 2.34.3
>
Re: [PATCH v12 0/1] vhost-vdpa: add support for vIOMMU
On Tue, 20 Dec 2022 at 22:33, Michael S. Tsirkin wrote: > > On Fri, Dec 09, 2022 at 09:08:04PM +0800, Cindy Lu wrote: > > These patches are to support vIOMMU in vdpa device > > Verified in vp_vdpa/vdpa_sim_net driverand intel_iommu > > virtio-iommu device > > Pls address comments and repost. > Hi Micheal There are some issues found while dpdk running with svq enable we are still working on it, and will post a new version after we address the bugs Thanks Cindy > > changes in V3 > > 1. Move function vfio_get_xlat_addr to memory.c > > 2. Use the existing memory listener, while the MR is > > iommu MR then call the function iommu_region_add/ > > iommu_region_del > > > > changes in V4 > > 1.make the comments in vfio_get_xlat_addr more general > > > > changes in V5 > > 1. Address the comments in the last version > > 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether > > the memory is backed by a discard manager. So the device can have its > > own warning. > > > > changes in V6 > > move the error_report for the unpopulated discard back to > > memeory_get_xlat_addr > > > > changes in V7 > > organize the error massage to avoid the duplicate information > > > > changes in V8 > > Organize the code follow the comments in the last version > > > > changes in V9 > > Organize the code follow the comments > > > > changes in V10 > > Address the comments > > > > changes in V11 > > Address the comments > > fix the crash found in test > > > > changes in V12 > > Address the comments, squash patch 1 into the next patch > > improve the code style issue > > > > Cindy Lu (1): > > vhost-vdpa: add support for vIOMMU > > > > hw/virtio/vhost-vdpa.c | 162 ++--- > > include/hw/virtio/vhost-vdpa.h | 10 ++ > > 2 files changed, 161 insertions(+), 11 deletions(-) > > > > -- > > 2.34.3 >
[PATCH v20 10/10] virtio-pci: add support for configure interrupt
Add process to handle the configure interrupt, The function's
logic is the same with vq interrupt.Add extra process to check
the configure interrupt
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 118 +++--
include/hw/virtio/virtio-pci.h | 4 +-
2 files changed, 102 insertions(+), 20 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index ec816ea367..3f00e91718 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -751,7 +751,8 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy *proxy,
int queue_no,
VirtQueue *vq;
if (queue_no == VIRTIO_CONFIG_IRQ_IDX) {
-return -1;
+*n = virtio_config_get_guest_notifier(vdev);
+*vector = vdev->config_vector;
} else {
if (!virtio_queue_get_num(vdev, queue_no)) {
return -1;
@@ -811,7 +812,7 @@ undo:
}
return ret;
}
-static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+static int kvm_virtio_pci_vector_vq_use(VirtIOPCIProxy *proxy, int nvqs)
{
int queue_no;
int ret = 0;
@@ -826,6 +827,10 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy
*proxy, int nvqs)
return ret;
}
+static int kvm_virtio_pci_vector_config_use(VirtIOPCIProxy *proxy)
+{
+return kvm_virtio_pci_vector_use_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
+}
static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
int queue_no)
@@ -850,7 +855,7 @@ static void
kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
kvm_virtio_pci_vq_vector_release(proxy, vector);
}
-static void kvm_virtio_pci_vector_release(VirtIOPCIProxy *proxy, int nvqs)
+static void kvm_virtio_pci_vector_vq_release(VirtIOPCIProxy *proxy, int nvqs)
{
int queue_no;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
@@ -863,6 +868,11 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
}
}
+static void kvm_virtio_pci_vector_config_release(VirtIOPCIProxy *proxy)
+{
+kvm_virtio_pci_vector_release_one(proxy, VIRTIO_CONFIG_IRQ_IDX);
+}
+
static int virtio_pci_one_vector_unmask(VirtIOPCIProxy *proxy,
unsigned int queue_no,
unsigned int vector,
@@ -944,9 +954,19 @@ static int virtio_pci_vector_unmask(PCIDevice *dev,
unsigned vector,
}
vq = virtio_vector_next_queue(vq);
}
-
+/* unmask config intr */
+if (vector == vdev->config_vector) {
+n = virtio_config_get_guest_notifier(vdev);
+ret = virtio_pci_one_vector_unmask(proxy, VIRTIO_CONFIG_IRQ_IDX,
vector,
+ msg, n);
+if (ret < 0) {
+goto undo_config;
+}
+}
return 0;
-
+undo_config:
+n = virtio_config_get_guest_notifier(vdev);
+virtio_pci_one_vector_mask(proxy, VIRTIO_CONFIG_IRQ_IDX, vector, n);
undo:
vq = virtio_vector_first_queue(vdev, vector);
while (vq && unmasked >= 0) {
@@ -980,6 +1000,11 @@ static void virtio_pci_vector_mask(PCIDevice *dev,
unsigned vector)
}
vq = virtio_vector_next_queue(vq);
}
+
+if (vector == vdev->config_vector) {
+n = virtio_config_get_guest_notifier(vdev);
+virtio_pci_one_vector_mask(proxy, VIRTIO_CONFIG_IRQ_IDX, vector, n);
+}
}
static void virtio_pci_vector_poll(PCIDevice *dev,
@@ -1011,6 +1036,34 @@ static void virtio_pci_vector_poll(PCIDevice *dev,
msix_set_pending(dev, vector);
}
}
+/* poll the config intr */
+ret = virtio_pci_get_notifier(proxy, VIRTIO_CONFIG_IRQ_IDX, ,
+ );
+if (ret < 0) {
+return;
+}
+if (vector < vector_start || vector >= vector_end ||
+!msix_is_masked(dev, vector)) {
+return;
+}
+if (k->guest_notifier_pending) {
+if (k->guest_notifier_pending(vdev, VIRTIO_CONFIG_IRQ_IDX)) {
+msix_set_pending(dev, vector);
+}
+} else if (event_notifier_test_and_clear(notifier)) {
+msix_set_pending(dev, vector);
+}
+}
+
+void virtio_pci_set_guest_notifier_fd_handler(VirtIODevice *vdev, VirtQueue
*vq,
+ int n, bool assign,
+ bool with_irqfd)
+{
+if (n == VIRTIO_CONFIG_IRQ_IDX) {
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+} else {
+virtio_queue_set_guest_notifier_fd_handler(vq, assign, with_irqfd);
+}
}
static int virtio_pci_set_guest_notifier(DeviceState *d, int n, bool assign,
@@ -1019,17 +1072,25 @@ static int virtio_pci_set_guest_notifier(DeviceState
*d, int n, bool assign,
VirtIOPCIProxy *proxy = to_virtio_pci_proxy(d);
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(
[PATCH v20 04/10] vhost: introduce new VhostOps vhost_set_config_call
This patch introduces new VhostOps vhost_set_config_call.
This function allows the qemu to set the config
event fd to kernel driver.
Signed-off-by: Cindy Lu
---
include/hw/virtio/vhost-backend.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/include/hw/virtio/vhost-backend.h
b/include/hw/virtio/vhost-backend.h
index eab46d7f0b..c5ab49051e 100644
--- a/include/hw/virtio/vhost-backend.h
+++ b/include/hw/virtio/vhost-backend.h
@@ -128,6 +128,8 @@ typedef int (*vhost_get_device_id_op)(struct vhost_dev
*dev, uint32_t *dev_id);
typedef bool (*vhost_force_iommu_op)(struct vhost_dev *dev);
+typedef int (*vhost_set_config_call_op)(struct vhost_dev *dev,
+ int fd);
typedef struct VhostOps {
VhostBackendType backend_type;
vhost_backend_init vhost_backend_init;
@@ -174,6 +176,7 @@ typedef struct VhostOps {
vhost_vq_get_addr_op vhost_vq_get_addr;
vhost_get_device_id_op vhost_get_device_id;
vhost_force_iommu_op vhost_force_iommu;
+vhost_set_config_call_op vhost_set_config_call;
} VhostOps;
int vhost_backend_update_device_iotlb(struct vhost_dev *dev,
--
2.34.3
[PATCH v20 07/10] vhost: add support for configure interrupt
Add functions to support configure interrupt.
The configure interrupt process will start in vhost_dev_start
and stop in vhost_dev_stop.
Also add the functions to support vhost_config_pending and
vhost_config_mask.
Signed-off-by: Cindy Lu
---
hw/virtio/vhost.c | 78 ++-
include/hw/virtio/vhost.h | 4 ++
2 files changed, 81 insertions(+), 1 deletion(-)
diff --git a/hw/virtio/vhost.c b/hw/virtio/vhost.c
index 7fb008bc9e..84dbb39e07 100644
--- a/hw/virtio/vhost.c
+++ b/hw/virtio/vhost.c
@@ -1596,7 +1596,68 @@ void vhost_virtqueue_mask(struct vhost_dev *hdev,
VirtIODevice *vdev, int n,
file.index = hdev->vhost_ops->vhost_get_vq_index(hdev, n);
r = hdev->vhost_ops->vhost_set_vring_call(hdev, );
if (r < 0) {
-VHOST_OPS_DEBUG(r, "vhost_set_vring_call failed");
+error_report("vhost_set_vring_call failed %d", -r);
+}
+}
+
+bool vhost_config_pending(struct vhost_dev *hdev)
+{
+assert(hdev->vhost_ops);
+if ((hdev->started == false) ||
+(hdev->vhost_ops->vhost_set_config_call == NULL)) {
+return false;
+}
+
+EventNotifier *notifier =
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier;
+return event_notifier_test_and_clear(notifier);
+}
+
+void vhost_config_mask(struct vhost_dev *hdev, VirtIODevice *vdev, bool mask)
+{
+int fd;
+int r;
+EventNotifier *notifier =
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier;
+EventNotifier *config_notifier = >config_notifier;
+assert(hdev->vhost_ops);
+
+if ((hdev->started == false) ||
+(hdev->vhost_ops->vhost_set_config_call == NULL)) {
+return;
+}
+if (mask) {
+assert(vdev->use_guest_notifier_mask);
+fd = event_notifier_get_fd(notifier);
+} else {
+fd = event_notifier_get_fd(config_notifier);
+}
+r = hdev->vhost_ops->vhost_set_config_call(hdev, fd);
+if (r < 0) {
+error_report("vhost_set_config_call failed %d", -r);
+}
+}
+
+static void vhost_stop_config_intr(struct vhost_dev *dev)
+{
+int fd = -1;
+assert(dev->vhost_ops);
+if (dev->vhost_ops->vhost_set_config_call) {
+dev->vhost_ops->vhost_set_config_call(dev, fd);
+}
+}
+
+static void vhost_start_config_intr(struct vhost_dev *dev)
+{
+int r;
+
+assert(dev->vhost_ops);
+int fd = event_notifier_get_fd(>vdev->config_notifier);
+if (dev->vhost_ops->vhost_set_config_call) {
+r = dev->vhost_ops->vhost_set_config_call(dev, fd);
+if (!r) {
+event_notifier_set(>vdev->config_notifier);
+}
}
}
@@ -1836,6 +1897,16 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
}
}
+r = event_notifier_init(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier, 0);
+if (r < 0) {
+return r;
+}
+event_notifier_test_and_clear(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier);
+if (!vdev->use_guest_notifier_mask) {
+vhost_config_mask(hdev, vdev, true);
+}
if (hdev->log_enabled) {
uint64_t log_base;
@@ -1874,6 +1945,7 @@ int vhost_dev_start(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
vhost_device_iotlb_miss(hdev, vq->used_phys, true);
}
}
+vhost_start_config_intr(hdev);
return 0;
fail_start:
if (vrings) {
@@ -1903,6 +1975,9 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
/* should only be called after backend is connected */
assert(hdev->vhost_ops);
+event_notifier_test_and_clear(
+>vqs[VHOST_QUEUE_NUM_CONFIG_INR].masked_config_notifier);
+event_notifier_test_and_clear(>config_notifier);
trace_vhost_dev_stop(hdev, vdev->name, vrings);
@@ -1925,6 +2000,7 @@ void vhost_dev_stop(struct vhost_dev *hdev, VirtIODevice
*vdev, bool vrings)
}
memory_listener_unregister(>iommu_listener);
}
+vhost_stop_config_intr(hdev);
vhost_log_put(hdev, true);
hdev->started = false;
vdev->vhost_started = false;
diff --git a/include/hw/virtio/vhost.h b/include/hw/virtio/vhost.h
index 67a6807fac..05bedb2416 100644
--- a/include/hw/virtio/vhost.h
+++ b/include/hw/virtio/vhost.h
@@ -33,6 +33,7 @@ struct vhost_virtqueue {
unsigned used_size;
EventNotifier masked_notifier;
EventNotifier error_notifier;
+EventNotifier masked_config_notifier;
struct vhost_dev *dev;
};
@@ -41,6 +42,7 @@ typedef unsigned long vhost_log_chunk_t;
#define VHOST_LOG_BITS (8 * sizeof(vhost_log_chunk_t))
#define VHOST_LOG_CHUNK (VHOST_LOG_PAGE * VHOST_LOG_BITS)
#define VHOST_INVALID_FEATURE_BIT (0xff)
+#define VHOST_QUEUE_NUM_CONFIG_INR 0
struct vhost_log {
unsigned lo
[PATCH v20 03/10] virtio-pci: decouple the single vector from the interrupt process
To reuse the interrupt process in configure interrupt
Need to decouple the single vector from the interrupt process.
We add new function kvm_virtio_pci_vector_use_one and _release_one.
These functions are used for the single vector, the whole process will
finish in the loop with vq number.
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 131 +++--
1 file changed, 73 insertions(+), 58 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index 52c7692fff..ec816ea367 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -699,7 +699,6 @@ static uint32_t virtio_read_config(PCIDevice *pci_dev,
}
static int kvm_virtio_pci_vq_vector_use(VirtIOPCIProxy *proxy,
-unsigned int queue_no,
unsigned int vector)
{
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
@@ -764,87 +763,103 @@ static int virtio_pci_get_notifier(VirtIOPCIProxy
*proxy, int queue_no,
return 0;
}
-static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+static int kvm_virtio_pci_vector_use_one(VirtIOPCIProxy *proxy, int queue_no)
{
+unsigned int vector;
+int ret;
+EventNotifier *n;
PCIDevice *dev = >pci_dev;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
-unsigned int vector;
-int ret, queue_no;
-EventNotifier *n;
-for (queue_no = 0; queue_no < nvqs; queue_no++) {
-if (!virtio_queue_get_num(vdev, queue_no)) {
-break;
-}
-ret = virtio_pci_get_notifier(proxy, queue_no, , );
-if (ret < 0) {
-break;
-}
-if (vector >= msix_nr_vectors_allocated(dev)) {
-continue;
-}
-ret = kvm_virtio_pci_vq_vector_use(proxy, queue_no, vector);
+
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+return ret;
+}
+if (vector >= msix_nr_vectors_allocated(dev)) {
+return 0;
+}
+ret = kvm_virtio_pci_vq_vector_use(proxy, vector);
+if (ret < 0) {
+goto undo;
+}
+/*
+ * If guest supports masking, set up irqfd now.
+ * Otherwise, delay until unmasked in the frontend.
+ */
+if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
+ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
if (ret < 0) {
+kvm_virtio_pci_vq_vector_release(proxy, vector);
goto undo;
}
-/* If guest supports masking, set up irqfd now.
- * Otherwise, delay until unmasked in the frontend.
- */
-if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
-if (ret < 0) {
-kvm_virtio_pci_vq_vector_release(proxy, vector);
-goto undo;
-}
-}
}
-return 0;
+return 0;
undo:
-while (--queue_no >= 0) {
-vector = virtio_queue_vector(vdev, queue_no);
-if (vector >= msix_nr_vectors_allocated(dev)) {
-continue;
+
+vector = virtio_queue_vector(vdev, queue_no);
+if (vector >= msix_nr_vectors_allocated(dev)) {
+return ret;
+}
+if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+return ret;
}
-if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = virtio_pci_get_notifier(proxy, queue_no, , );
-if (ret < 0) {
-break;
-}
-kvm_virtio_pci_irqfd_release(proxy, n, vector);
+kvm_virtio_pci_irqfd_release(proxy, n, vector);
+}
+return ret;
+}
+static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
+{
+int queue_no;
+int ret = 0;
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+
+for (queue_no = 0; queue_no < nvqs; queue_no++) {
+if (!virtio_queue_get_num(vdev, queue_no)) {
+return -1;
}
-kvm_virtio_pci_vq_vector_release(proxy, vector);
+ret = kvm_virtio_pci_vector_use_one(proxy, queue_no);
}
return ret;
}
-static void kvm_virtio_pci_vector_release(VirtIOPCIProxy *proxy, int nvqs)
+
+static void kvm_virtio_pci_vector_release_one(VirtIOPCIProxy *proxy,
+ int queue_no)
{
-PCIDevice *dev = >pci_dev;
VirtIODevice *vdev = virtio_bus_get_device(>bus);
unsigned int vector;
-int queue_no;
-VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
EventNotifier *n;
-int ret ;
+int ret;
+VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
+PCIDevice *dev = >pci_dev;
[PATCH v20 09/10] virtio-mmio: add support for configure interrupt
Add configure interrupt support in virtio-mmio bus.
add function to set configure guest notifier.
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-mmio.c | 27 +++
1 file changed, 27 insertions(+)
diff --git a/hw/virtio/virtio-mmio.c b/hw/virtio/virtio-mmio.c
index d240efef97..103260ec15 100644
--- a/hw/virtio/virtio-mmio.c
+++ b/hw/virtio/virtio-mmio.c
@@ -670,7 +670,30 @@ static int virtio_mmio_set_guest_notifier(DeviceState *d,
int n, bool assign,
return 0;
}
+static int virtio_mmio_set_config_guest_notifier(DeviceState *d, bool assign,
+ bool with_irqfd)
+{
+VirtIOMMIOProxy *proxy = VIRTIO_MMIO(d);
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+VirtioDeviceClass *vdc = VIRTIO_DEVICE_GET_CLASS(vdev);
+EventNotifier *notifier = virtio_config_get_guest_notifier(vdev);
+int r = 0;
+if (assign) {
+r = event_notifier_init(notifier, 0);
+if (r < 0) {
+return r;
+}
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+} else {
+virtio_config_set_guest_notifier_fd_handler(vdev, assign, with_irqfd);
+event_notifier_cleanup(notifier);
+}
+if (vdc->guest_notifier_mask && vdev->use_guest_notifier_mask) {
+vdc->guest_notifier_mask(vdev, VIRTIO_CONFIG_IRQ_IDX, !assign);
+}
+return r;
+}
static int virtio_mmio_set_guest_notifiers(DeviceState *d, int nvqs,
bool assign)
{
@@ -692,6 +715,10 @@ static int virtio_mmio_set_guest_notifiers(DeviceState *d,
int nvqs,
goto assign_error;
}
}
+r = virtio_mmio_set_config_guest_notifier(d, assign, with_irqfd);
+if (r < 0) {
+goto assign_error;
+}
return 0;
--
2.34.3
[PATCH v20 00/10] vhost-vdpa: add support for configure interrupt
These patches introduced the support for configure interrupt These codes are tested on x86_64 and aarch64 platforms. the tested on vp-vdpa/vdpa_sim_net /vhost/vhost_user/testpmd, with/without irqfd. Tested in virtio-pci bus and virtio-mmio bus Change in v2: Add support for virtio-mmio bus active the notifier while the backend support configure interrupt misc fixes from v1 Change in v3 fix the coding style problems Change in v4 misc fixes from v3 merge the set_config_notifier to set_guest_notifier when vdpa start, check the feature by VIRTIO_NET_F_STATUS Change in v5 misc fixes from v4 split the code to introduce configure interrupt type and the callback function will init the configure interrupt in all virtio-pci and virtio-mmio bus, but will only active while using vhost-vdpa driver Change in v6 misc fixes from v5 decouple vq from interrupt setting and misc process fix the bug in virtio_net_handle_rx Change in v7 misc fixes from v6 decouple vq from interrupt setting and misc process decouple vq from vector use/release process decouple vq from set notifier fd handler process move config_notifier and masked_config_notifier to VirtIODevice fix the bug in virtio_net_handle_rx, add more information add VIRTIO_CONFIG_IRQ_IDX as the queue number for configure interrupt Change in v8 misc fixes from v7 decouple vq from interrupt setting and misc process decouple vq from vector use/release process decouple vq from set notifier fd handler process move the vhost configure interrupt to vhost_net Change in v9 misc fixes from v8 address the comments from v8 Change in v10 fix the hang issue in qtest address the comments from v9 Change in v11 fix the crash in aarch64 plateform. fix the crash upstream reported Change in v12 fix the typo and the comments changes in v13 re-send the patches by git-publish changes in v14 rebased the code based on upstream changes in v15 rebased the code based on upstream changes in v16 resend the patch, seems there not send successfully changes in v18 fix the crash in qtest Because hw/virtio/vhost-user-gpio.c is a new device,So I missed to add the VIRTIO_CONFIG_IRQ_IDX check in notifier_mask changes in v19 fix the crash in qtest rebased the code based on upstream Cindy Lu (10): virtio: introduce macro VIRTIO_CONFIG_IRQ_IDX virtio-pci: decouple notifier from interrupt process virtio-pci: decouple the single vector from the interrupt process vhost: introduce new VhostOps vhost_set_config_call vhost-vdpa: add support for config interrupt virtio: add support for configure interrupt vhost: add support for configure interrupt virtio-net: add support for configure interrupt virtio-mmio: add support for configure interrupt virtio-pci: add support for configure interrupt hw/display/vhost-user-gpu.c | 18 ++ hw/net/vhost_net.c| 9 + hw/net/virtio-net.c | 22 ++- hw/virtio/trace-events| 1 + hw/virtio/vhost-user-fs.c | 18 ++ hw/virtio/vhost-user-gpio.c | 10 + hw/virtio/vhost-vdpa.c| 8 + hw/virtio/vhost-vsock-common.c| 18 ++ hw/virtio/vhost.c | 78 +++- hw/virtio/virtio-crypto.c | 18 ++ hw/virtio/virtio-mmio.c | 27 +++ hw/virtio/virtio-pci.c| 291 +- hw/virtio/virtio.c| 29 +++ include/hw/virtio/vhost-backend.h | 3 + include/hw/virtio/vhost.h | 4 + include/hw/virtio/virtio-pci.h| 4 +- include/hw/virtio/virtio.h| 7 + include/net/vhost_net.h | 2 + 18 files changed, 478 insertions(+), 89 deletions(-) -- 2.34.3
[PATCH v20 08/10] virtio-net: add support for configure interrupt
Add functions to support configure interrupt in virtio_net
Add the functions to support vhost_net_config_pending
and vhost_net_config_mask.
Signed-off-by: Cindy Lu
---
hw/net/vhost_net.c | 9 +
hw/net/virtio-net.c | 4 ++--
include/net/vhost_net.h | 2 ++
3 files changed, 13 insertions(+), 2 deletions(-)
diff --git a/hw/net/vhost_net.c b/hw/net/vhost_net.c
index 043058ff43..6a55f5a473 100644
--- a/hw/net/vhost_net.c
+++ b/hw/net/vhost_net.c
@@ -478,6 +478,15 @@ void vhost_net_virtqueue_mask(VHostNetState *net,
VirtIODevice *dev,
vhost_virtqueue_mask(>dev, dev, idx, mask);
}
+bool vhost_net_config_pending(VHostNetState *net)
+{
+return vhost_config_pending(>dev);
+}
+
+void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool mask)
+{
+vhost_config_mask(>dev, dev, mask);
+}
VHostNetState *get_vhost_net(NetClientState *nc)
{
VHostNetState *vhost_net = 0;
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index bee35d6f9f..ec974f7a76 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3323,7 +3323,7 @@ static bool
virtio_net_guest_notifier_pending(VirtIODevice *vdev, int idx)
*/
if (idx == VIRTIO_CONFIG_IRQ_IDX) {
-return false;
+return vhost_net_config_pending(get_vhost_net(nc->peer));
}
return vhost_net_virtqueue_pending(get_vhost_net(nc->peer), idx);
}
@@ -3355,9 +3355,9 @@ static void virtio_net_guest_notifier_mask(VirtIODevice
*vdev, int idx,
*/
if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+vhost_net_config_mask(get_vhost_net(nc->peer), vdev, mask);
return;
}
-
vhost_net_virtqueue_mask(get_vhost_net(nc->peer), vdev, idx, mask);
}
diff --git a/include/net/vhost_net.h b/include/net/vhost_net.h
index 40b9a40074..dbbd0dc04e 100644
--- a/include/net/vhost_net.h
+++ b/include/net/vhost_net.h
@@ -39,6 +39,8 @@ int vhost_net_set_config(struct vhost_net *net, const uint8_t
*data,
bool vhost_net_virtqueue_pending(VHostNetState *net, int n);
void vhost_net_virtqueue_mask(VHostNetState *net, VirtIODevice *dev,
int idx, bool mask);
+bool vhost_net_config_pending(VHostNetState *net);
+void vhost_net_config_mask(VHostNetState *net, VirtIODevice *dev, bool mask);
int vhost_net_notify_migration_done(VHostNetState *net, char* mac_addr);
VHostNetState *get_vhost_net(NetClientState *nc);
--
2.34.3
[PATCH v20 06/10] virtio: add support for configure interrupt
Add the functions to support the configure interrupt in virtio
The function virtio_config_guest_notifier_read will notify the
guest if there is an configure interrupt.
The function virtio_config_set_guest_notifier_fd_handler is
to set the fd hander for the notifier
Signed-off-by: Cindy Lu
---
hw/virtio/virtio.c | 29 +
include/hw/virtio/virtio.h | 4
2 files changed, 33 insertions(+)
diff --git a/hw/virtio/virtio.c b/hw/virtio/virtio.c
index eb6347ab5d..34e9c5d141 100644
--- a/hw/virtio/virtio.c
+++ b/hw/virtio/virtio.c
@@ -4012,7 +4012,14 @@ static void
virtio_queue_guest_notifier_read(EventNotifier *n)
virtio_irq(vq);
}
}
+static void virtio_config_guest_notifier_read(EventNotifier *n)
+{
+VirtIODevice *vdev = container_of(n, VirtIODevice, config_notifier);
+if (event_notifier_test_and_clear(n)) {
+virtio_notify_config(vdev);
+}
+}
void virtio_queue_set_guest_notifier_fd_handler(VirtQueue *vq, bool assign,
bool with_irqfd)
{
@@ -4029,6 +4036,23 @@ void
virtio_queue_set_guest_notifier_fd_handler(VirtQueue *vq, bool assign,
}
}
+void virtio_config_set_guest_notifier_fd_handler(VirtIODevice *vdev,
+ bool assign, bool with_irqfd)
+{
+EventNotifier *n;
+n = >config_notifier;
+if (assign && !with_irqfd) {
+event_notifier_set_handler(n, virtio_config_guest_notifier_read);
+} else {
+event_notifier_set_handler(n, NULL);
+}
+if (!assign) {
+/* Test and clear notifier before closing it,*/
+/* in case poll callback didn't have time to run. */
+virtio_config_guest_notifier_read(n);
+}
+}
+
EventNotifier *virtio_queue_get_guest_notifier(VirtQueue *vq)
{
return >guest_notifier;
@@ -4109,6 +4133,11 @@ EventNotifier *virtio_queue_get_host_notifier(VirtQueue
*vq)
return >host_notifier;
}
+EventNotifier *virtio_config_get_guest_notifier(VirtIODevice *vdev)
+{
+return >config_notifier;
+}
+
void virtio_queue_set_host_notifier_enabled(VirtQueue *vq, bool enabled)
{
vq->host_notifier_enabled = enabled;
diff --git a/include/hw/virtio/virtio.h b/include/hw/virtio/virtio.h
index 1f4a41b958..9c3a4642f2 100644
--- a/include/hw/virtio/virtio.h
+++ b/include/hw/virtio/virtio.h
@@ -138,6 +138,7 @@ struct VirtIODevice
AddressSpace *dma_as;
QLIST_HEAD(, VirtQueue) *vector_queues;
QTAILQ_ENTRY(VirtIODevice) next;
+EventNotifier config_notifier;
};
struct VirtioDeviceClass {
@@ -360,6 +361,9 @@ void
virtio_queue_aio_attach_host_notifier_no_poll(VirtQueue *vq, AioContext *ct
void virtio_queue_aio_detach_host_notifier(VirtQueue *vq, AioContext *ctx);
VirtQueue *virtio_vector_first_queue(VirtIODevice *vdev, uint16_t vector);
VirtQueue *virtio_vector_next_queue(VirtQueue *vq);
+EventNotifier *virtio_config_get_guest_notifier(VirtIODevice *vdev);
+void virtio_config_set_guest_notifier_fd_handler(VirtIODevice *vdev,
+ bool assign, bool with_irqfd);
static inline void virtio_add_feature(uint64_t *features, unsigned int fbit)
{
--
2.34.3
[PATCH v20 02/10] virtio-pci: decouple notifier from interrupt process
To reuse the notifier process. We add the virtio_pci_get_notifier
to get the notifier and vector. The INPUT for this function is IDX,
The OUTPUT is the notifier and the vector
Signed-off-by: Cindy Lu
---
hw/virtio/virtio-pci.c | 88 +++---
1 file changed, 57 insertions(+), 31 deletions(-)
diff --git a/hw/virtio/virtio-pci.c b/hw/virtio/virtio-pci.c
index a1c9dfa7bb..52c7692fff 100644
--- a/hw/virtio/virtio-pci.c
+++ b/hw/virtio/virtio-pci.c
@@ -728,29 +728,41 @@ static void
kvm_virtio_pci_vq_vector_release(VirtIOPCIProxy *proxy,
}
static int kvm_virtio_pci_irqfd_use(VirtIOPCIProxy *proxy,
- unsigned int queue_no,
+ EventNotifier *n,
unsigned int vector)
{
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
-VirtIODevice *vdev = virtio_bus_get_device(>bus);
-VirtQueue *vq = virtio_get_queue(vdev, queue_no);
-EventNotifier *n = virtio_queue_get_guest_notifier(vq);
return kvm_irqchip_add_irqfd_notifier_gsi(kvm_state, n, NULL, irqfd->virq);
}
static void kvm_virtio_pci_irqfd_release(VirtIOPCIProxy *proxy,
- unsigned int queue_no,
+ EventNotifier *n ,
unsigned int vector)
{
-VirtIODevice *vdev = virtio_bus_get_device(>bus);
-VirtQueue *vq = virtio_get_queue(vdev, queue_no);
-EventNotifier *n = virtio_queue_get_guest_notifier(vq);
VirtIOIRQFD *irqfd = >vector_irqfd[vector];
int ret;
ret = kvm_irqchip_remove_irqfd_notifier_gsi(kvm_state, n, irqfd->virq);
assert(ret == 0);
}
+static int virtio_pci_get_notifier(VirtIOPCIProxy *proxy, int queue_no,
+ EventNotifier **n, unsigned int *vector)
+{
+VirtIODevice *vdev = virtio_bus_get_device(>bus);
+VirtQueue *vq;
+
+if (queue_no == VIRTIO_CONFIG_IRQ_IDX) {
+return -1;
+} else {
+if (!virtio_queue_get_num(vdev, queue_no)) {
+return -1;
+}
+*vector = virtio_queue_vector(vdev, queue_no);
+vq = virtio_get_queue(vdev, queue_no);
+*n = virtio_queue_get_guest_notifier(vq);
+}
+return 0;
+}
static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy, int nvqs)
{
@@ -759,12 +771,15 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy
*proxy, int nvqs)
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
unsigned int vector;
int ret, queue_no;
-
+EventNotifier *n;
for (queue_no = 0; queue_no < nvqs; queue_no++) {
if (!virtio_queue_get_num(vdev, queue_no)) {
break;
}
-vector = virtio_queue_vector(vdev, queue_no);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
if (vector >= msix_nr_vectors_allocated(dev)) {
continue;
}
@@ -776,7 +791,7 @@ static int kvm_virtio_pci_vector_use(VirtIOPCIProxy *proxy,
int nvqs)
* Otherwise, delay until unmasked in the frontend.
*/
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-ret = kvm_virtio_pci_irqfd_use(proxy, queue_no, vector);
+ret = kvm_virtio_pci_irqfd_use(proxy, n, vector);
if (ret < 0) {
kvm_virtio_pci_vq_vector_release(proxy, vector);
goto undo;
@@ -792,7 +807,11 @@ undo:
continue;
}
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-kvm_virtio_pci_irqfd_release(proxy, queue_no, vector);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
+kvm_virtio_pci_irqfd_release(proxy, n, vector);
}
kvm_virtio_pci_vq_vector_release(proxy, vector);
}
@@ -806,12 +825,16 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
unsigned int vector;
int queue_no;
VirtioDeviceClass *k = VIRTIO_DEVICE_GET_CLASS(vdev);
-
+EventNotifier *n;
+int ret ;
for (queue_no = 0; queue_no < nvqs; queue_no++) {
if (!virtio_queue_get_num(vdev, queue_no)) {
break;
}
-vector = virtio_queue_vector(vdev, queue_no);
+ret = virtio_pci_get_notifier(proxy, queue_no, , );
+if (ret < 0) {
+break;
+}
if (vector >= msix_nr_vectors_allocated(dev)) {
continue;
}
@@ -819,21 +842,20 @@ static void kvm_virtio_pci_vector_release(VirtIOPCIProxy
*proxy, int nvqs)
* Otherwise, it was cleaned when masked in the frontend.
*/
if (vdev->use_guest_notifier_mask && k->guest_notifier_mask) {
-kvm_virtio_pci_irqfd_relea
[PATCH v20 01/10] virtio: introduce macro VIRTIO_CONFIG_IRQ_IDX
To support configure interrupt for vhost-vdpa
Introduce VIRTIO_CONFIG_IRQ_IDX -1 as configure interrupt's queue index,
Then we can reuse the functions guest_notifier_mask and guest_notifier_pending.
Add the check of queue index in these drivers, if the driver does not support
configure interrupt, the function will just return
Signed-off-by: Cindy Lu
---
hw/display/vhost-user-gpu.c| 18 ++
hw/net/virtio-net.c| 22 --
hw/virtio/vhost-user-fs.c | 18 ++
hw/virtio/vhost-user-gpio.c| 10 ++
hw/virtio/vhost-vsock-common.c | 18 ++
hw/virtio/virtio-crypto.c | 18 ++
include/hw/virtio/virtio.h | 3 +++
7 files changed, 105 insertions(+), 2 deletions(-)
diff --git a/hw/display/vhost-user-gpu.c b/hw/display/vhost-user-gpu.c
index 19c0e20103..4380a5e672 100644
--- a/hw/display/vhost-user-gpu.c
+++ b/hw/display/vhost-user-gpu.c
@@ -486,6 +486,15 @@ vhost_user_gpu_guest_notifier_pending(VirtIODevice *vdev,
int idx)
{
VhostUserGPU *g = VHOST_USER_GPU(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_virtqueue_pending(>vhost->dev, idx);
}
@@ -494,6 +503,15 @@ vhost_user_gpu_guest_notifier_mask(VirtIODevice *vdev, int
idx, bool mask)
{
VhostUserGPU *g = VHOST_USER_GPU(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
vhost_virtqueue_mask(>vhost->dev, vdev, idx, mask);
}
diff --git a/hw/net/virtio-net.c b/hw/net/virtio-net.c
index aba12759d5..bee35d6f9f 100644
--- a/hw/net/virtio-net.c
+++ b/hw/net/virtio-net.c
@@ -3316,6 +3316,15 @@ static bool
virtio_net_guest_notifier_pending(VirtIODevice *vdev, int idx)
} else {
nc = qemu_get_subqueue(n->nic, vq2q(idx));
}
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return false
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_net_virtqueue_pending(get_vhost_net(nc->peer), idx);
}
@@ -3339,8 +3348,17 @@ static void virtio_net_guest_notifier_mask(VirtIODevice
*vdev, int idx,
} else {
nc = qemu_get_subqueue(n->nic, vq2q(idx));
}
-vhost_net_virtqueue_mask(get_vhost_net(nc->peer),
- vdev, idx, mask);
+/*
+ *Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
+
+vhost_net_virtqueue_mask(get_vhost_net(nc->peer), vdev, idx, mask);
}
static void virtio_net_set_config_size(VirtIONet *n, uint64_t host_features)
diff --git a/hw/virtio/vhost-user-fs.c b/hw/virtio/vhost-user-fs.c
index d97b179e6f..f5049735ac 100644
--- a/hw/virtio/vhost-user-fs.c
+++ b/hw/virtio/vhost-user-fs.c
@@ -159,6 +159,15 @@ static void vuf_guest_notifier_mask(VirtIODevice *vdev,
int idx,
{
VHostUserFS *fs = VHOST_USER_FS(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return;
+}
vhost_virtqueue_mask(>vhost_dev, vdev, idx, mask);
}
@@ -166,6 +175,15 @@ static bool vuf_guest_notifier_pending(VirtIODevice *vdev,
int idx)
{
VHostUserFS *fs = VHOST_USER_FS(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function will return
+ */
+
+if (idx == VIRTIO_CONFIG_IRQ_IDX) {
+return false;
+}
return vhost_virtqueue_pending(>vhost_dev, idx);
}
diff --git a/hw/virtio/vhost-user-gpio.c b/hw/virtio/vhost-user-gpio.c
index b7b82a1099..fe3da32c74 100644
--- a/hw/virtio/vhost-user-gpio.c
+++ b/hw/virtio/vhost-user-gpio.c
@@ -191,6 +191,16 @@ static void vu_gpio_guest_notifier_mask(VirtIODevice
*vdev, int idx, bool mask)
{
VHostUserGPIO *gpio = VHOST_USER_GPIO(vdev);
+/*
+ * Add the check for configure interrupt, Use VIRTIO_CONFIG_IRQ_IDX -1
+ * as the Marco of configure interrupt's IDX, If this driver does not
+ * support, the function wi
[PATCH v20 05/10] vhost-vdpa: add support for config interrupt
Add new call back function in vhost-vdpa, The function
vhost_set_config_call can set the event fd to kernel.
This function will be called in the vhost_dev_start
and vhost_dev_stop
Signed-off-by: Cindy Lu
---
hw/virtio/trace-events | 1 +
hw/virtio/vhost-vdpa.c | 8
2 files changed, 9 insertions(+)
diff --git a/hw/virtio/trace-events b/hw/virtio/trace-events
index 14fc5b9bb2..46f2faf04e 100644
--- a/hw/virtio/trace-events
+++ b/hw/virtio/trace-events
@@ -62,6 +62,7 @@ vhost_vdpa_get_features(void *dev, uint64_t features) "dev:
%p features: 0x%"PRI
vhost_vdpa_set_owner(void *dev) "dev: %p"
vhost_vdpa_vq_get_addr(void *dev, void *vq, uint64_t desc_user_addr, uint64_t
avail_user_addr, uint64_t used_user_addr) "dev: %p vq: %p desc_user_addr:
0x%"PRIx64" avail_user_addr: 0x%"PRIx64" used_user_addr: 0x%"PRIx64
vhost_vdpa_get_iova_range(void *dev, uint64_t first, uint64_t last) "dev: %p
first: 0x%"PRIx64" last: 0x%"PRIx64
+vhost_vdpa_set_config_call(void *dev, int fd)"dev: %p fd: %d"
# virtio.c
virtqueue_alloc_element(void *elem, size_t sz, unsigned in_num, unsigned
out_num) "elem %p size %zd in_num %u out_num %u"
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 2b3920c2a1..7f3e3dd740 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -893,6 +893,13 @@ static int vhost_vdpa_set_vring_ready(struct vhost_dev
*dev)
return 0;
}
+static int vhost_vdpa_set_config_call(struct vhost_dev *dev,
+ int fd)
+{
+trace_vhost_vdpa_set_config_call(dev, fd);
+return vhost_vdpa_call(dev, VHOST_VDPA_SET_CONFIG_CALL, );
+}
+
static void vhost_vdpa_dump_config(struct vhost_dev *dev, const uint8_t
*config,
uint32_t config_len)
{
@@ -1450,4 +1457,5 @@ const VhostOps vdpa_ops = {
.vhost_get_device_id = vhost_vdpa_get_device_id,
.vhost_vq_get_addr = vhost_vdpa_vq_get_addr,
.vhost_force_iommu = vhost_vdpa_force_iommu,
+.vhost_set_config_call = vhost_vdpa_set_config_call,
};
--
2.34.3
[PATCH v12 1/1] vhost-vdpa: add support for vIOMMU
1.Skip the check in vhost_vdpa_listener_skipped_section() while
MR is IOMMU, Move this check to vhost_vdpa_iommu_map_notify()
2.Add support for vIOMMU.
Add the new function to deal with iommu MR.
- during iommu_region_add register a specific IOMMU notifier,
and store all notifiers in a list.
- during iommu_region_del, compare and delete the IOMMU notifier from the list
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 162 ++---
include/hw/virtio/vhost-vdpa.h | 10 ++
2 files changed, 161 insertions(+), 11 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 7468e44b87..2b3920c2a1 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -60,15 +61,22 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, Sometimes the section will be larger than iova_max
+ * but the memory that actually mapping is smaller, So skip the check
+ * here. Will add the check in vhost_vdpa_iommu_map_notify,
+ *There is the real size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
-
return false;
}
@@ -173,6 +181,115 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+Int128 llend;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+/* check if RAM section out of device range */
+llend = int128_add(int128_makes64(iotlb->addr_mask), int128_makes64(iova));
+if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ v->iova_range.last, int128_get64(llend));
+return;
+}
+
+vhost_vdpa_iotlb_batch_begin_once(v);
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+
+ret = vhost_vdpa_dma_map(v, iova, iotlb->addr_mask + 1, vaddr,
+read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, iova, iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+Int128 end;
+int iommu_idx;
+IOMMUMemoryRegion *iommu_mr;
+int ret;
+
+iommu_mr = IOMMU_MEMORY_REGION(section->mr);
+
+iommu = g_malloc0(sizeof(*iommu));
+end = int128_add(int128_make64(section->offset_within_region),
+section->si
[PATCH v12 0/1] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device Verified in vp_vdpa/vdpa_sim_net driverand intel_iommu virtio-iommu device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments fix the crash found in test changes in V12 Address the comments, squash patch 1 into the next patch improve the code style issue Cindy Lu (1): vhost-vdpa: add support for vIOMMU hw/virtio/vhost-vdpa.c | 162 ++--- include/hw/virtio/vhost-vdpa.h | 10 ++ 2 files changed, 161 insertions(+), 11 deletions(-) -- 2.34.3
Re: [PATCH v11 1/2] vhost-vdpa: Skip the range check while MR is IOMMU
On Thu, 8 Dec 2022 at 15:42, Jason Wang wrote:
>
> On Wed, Nov 30, 2022 at 1:33 PM Cindy Lu wrote:
> >
> > Skip the check in vhost_vdpa_listener_skipped_section() while
> > MR is IOMMU, Move this check to vhost_vdpa_iommu_map_notify()
> >
> > Signed-off-by: Cindy Lu
> > ---
> > hw/virtio/vhost-vdpa.c | 21 ++---
> > 1 file changed, 14 insertions(+), 7 deletions(-)
> >
> > diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
> > index 3ff9ce3501..f0e9963d19 100644
> > --- a/hw/virtio/vhost-vdpa.c
> > +++ b/hw/virtio/vhost-vdpa.c
> > @@ -60,15 +60,22 @@ static bool
> > vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
> > iova_min, section->offset_within_address_space);
> > return true;
> > }
> > +/*
> > + * While using vIOMMU, Sometimes the section will be larger than
> > iova_max
> > + * but the memory that actually mapping is smaller, So skip the check
> > + * here. Will add the check in vhost_vdpa_iommu_map_notify,
> > + *There is the real size that maps to the kernel
> > + */
> >
>
> I suggest squashing this into the next patch since we haven't
> implemented vhost_vdpa_iommu_map_notify() yet.
>
> Thanks
>
Sure, will do
Thanks
Cindy
> > -llend = vhost_vdpa_section_end(section);
> > -if (int128_gt(llend, int128_make64(iova_max))) {
> > -error_report("RAM section out of device range (max=0x%" PRIx64
> > - ", end addr=0x%" PRIx64 ")",
> > - iova_max, int128_get64(llend));
> > -return true;
> > +if (!memory_region_is_iommu(section->mr)) {
> > +llend = vhost_vdpa_section_end(section);
> > +if (int128_gt(llend, int128_make64(iova_max))) {
> > +error_report("RAM section out of device range (max=0x%" PRIx64
> > + ", end addr=0x%" PRIx64 ")",
> > + iova_max, int128_get64(llend));
> > +return true;
> > +}
> > }
> > -
> > return false;
> > }
> >
> > --
> > 2.34.3
> >
>
Re: [PATCH] vhost: Not return fail while the device does not support send_iotlb_msg
On Thu, 1 Dec 2022 at 16:49, Jason Wang wrote: > > On Wed, Nov 30, 2022 at 4:11 PM Cindy Lu wrote: > > > > Some device does not support vhost_send_device_iotlb_msg() > > such as vDPA device, which is as expected. So we should not > > return fail here. > > Please explain in which case you may hit the -ENODEV and what's the > side effect of this. > > Thanks > this issue was found during the test of virtio-iommu the step is 1. while load the VM with qemu -device virtio-iommu-pci \ -device virtio-net-pci,netdev=vhost-vdpa0,disable-legacy=on,disable-modern=off,iommu_platform=on\ -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa0\ 2. the guest vm's CMDLINE //proc/cmdline don't have iommu=pt there will be a lot error message during loading VM/runing traffic ysteqemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb mqemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb dqemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb -resolved.…e - Network Name Resolution...qemu-system-x86_64: Fail to invalidate device iotlb qemu-system-x86_64: Fail to invalidate device iotlb . and the vdpa port in guest VM doesn't working well With this fix the guest VM load without error message and the vdpa port working correctly at the same setting [root@ubuntunew ~]# ping 111.1.1.2 PING 111.1.1.2 (111.1.1.2) 56(84) bytes of data. 64 bytes from 111.1.1.2: icmp_seq=1 ttl=64 time=25.0 ms 64 bytes from 111.1.1.2: icmp_seq=2 ttl=64 time=22.0 ms 64 bytes from 111.1.1.2: icmp_seq=3 ttl=64 time=24.3 ms Thansk Cindy > > > > Signed-off-by: Cindy Lu > > --- > > hw/virtio/vhost-backend.c | 4 ++-- > > 1 file changed, 2 insertions(+), 2 deletions(-) > > > > diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c > > index 8e581575c9..9321ed9031 100644 > > --- a/hw/virtio/vhost-backend.c > > +++ b/hw/virtio/vhost-backend.c > > @@ -360,7 +360,7 @@ int vhost_backend_update_device_iotlb(struct vhost_dev > > *dev, > > if (dev->vhost_ops && dev->vhost_ops->vhost_send_device_iotlb_msg) > > return dev->vhost_ops->vhost_send_device_iotlb_msg(dev, ); > > > > -return -ENODEV; > > +return 0; > > } > > > > int vhost_backend_invalidate_device_iotlb(struct vhost_dev *dev, > > @@ -375,7 +375,7 @@ int vhost_backend_invalidate_device_iotlb(struct > > vhost_dev *dev, > > if (dev->vhost_ops && dev->vhost_ops->vhost_send_device_iotlb_msg) > > return dev->vhost_ops->vhost_send_device_iotlb_msg(dev, ); > > > > -return -ENODEV; > > +return 0; > > } > > > > int vhost_backend_handle_iotlb_msg(struct vhost_dev *dev, > > -- > > 2.34.3 > > >
[PATCH] vhost: Not return fail while the device does not support send_iotlb_msg
Some device does not support vhost_send_device_iotlb_msg() such as vDPA device, which is as expected. So we should not return fail here. Signed-off-by: Cindy Lu --- hw/virtio/vhost-backend.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/hw/virtio/vhost-backend.c b/hw/virtio/vhost-backend.c index 8e581575c9..9321ed9031 100644 --- a/hw/virtio/vhost-backend.c +++ b/hw/virtio/vhost-backend.c @@ -360,7 +360,7 @@ int vhost_backend_update_device_iotlb(struct vhost_dev *dev, if (dev->vhost_ops && dev->vhost_ops->vhost_send_device_iotlb_msg) return dev->vhost_ops->vhost_send_device_iotlb_msg(dev, ); -return -ENODEV; +return 0; } int vhost_backend_invalidate_device_iotlb(struct vhost_dev *dev, @@ -375,7 +375,7 @@ int vhost_backend_invalidate_device_iotlb(struct vhost_dev *dev, if (dev->vhost_ops && dev->vhost_ops->vhost_send_device_iotlb_msg) return dev->vhost_ops->vhost_send_device_iotlb_msg(dev, ); -return -ENODEV; +return 0; } int vhost_backend_handle_iotlb_msg(struct vhost_dev *dev, -- 2.34.3
[PATCH v11 0/2] vhost-vdpa: add support for vIOMMU
These patches are to support vIOMMU in vdpa device changes in V3 1. Move function vfio_get_xlat_addr to memory.c 2. Use the existing memory listener, while the MR is iommu MR then call the function iommu_region_add/ iommu_region_del changes in V4 1.make the comments in vfio_get_xlat_addr more general changes in V5 1. Address the comments in the last version 2. Add a new arg in the function vfio_get_xlat_addr, which shows whether the memory is backed by a discard manager. So the device can have its own warning. changes in V6 move the error_report for the unpopulated discard back to memeory_get_xlat_addr changes in V7 organize the error massage to avoid the duplicate information changes in V8 Organize the code follow the comments in the last version changes in V9 Organize the code follow the comments changes in V10 Address the comments changes in V11 Address the comments Fix the crash found in test Cindy Lu (2): vhost-vdpa: Skip the range check while MR is IOMMU vhost-vdpa: add support for vIOMMU hw/virtio/vhost-vdpa.c | 168 ++--- include/hw/virtio/vhost-vdpa.h | 10 ++ 2 files changed, 167 insertions(+), 11 deletions(-) -- 2.34.3
[PATCH v11 1/2] vhost-vdpa: Skip the range check while MR is IOMMU
Skip the check in vhost_vdpa_listener_skipped_section() while
MR is IOMMU, Move this check to vhost_vdpa_iommu_map_notify()
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 21 ++---
1 file changed, 14 insertions(+), 7 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 3ff9ce3501..f0e9963d19 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -60,15 +60,22 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
iova_min, section->offset_within_address_space);
return true;
}
+/*
+ * While using vIOMMU, Sometimes the section will be larger than iova_max
+ * but the memory that actually mapping is smaller, So skip the check
+ * here. Will add the check in vhost_vdpa_iommu_map_notify,
+ *There is the real size that maps to the kernel
+ */
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
+if (!memory_region_is_iommu(section->mr)) {
+llend = vhost_vdpa_section_end(section);
+if (int128_gt(llend, int128_make64(iova_max))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ iova_max, int128_get64(llend));
+return true;
+}
}
-
return false;
}
--
2.34.3
[PATCH v11 2/2] vhost-vdpa: add support for vIOMMU
Add support for vIOMMU. add the new function to deal with iommu MR.
- during iommu_region_add register a specific IOMMU notifier,
and store all notifiers in a list.
- during iommu_region_del, compare and delete the IOMMU notifier from the list
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 147 -
include/hw/virtio/vhost-vdpa.h | 10 +++
2 files changed, 153 insertions(+), 4 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index f0e9963d19..4137094fef 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -180,6 +181,119 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+Int128 llend;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+/* check if RAM section out of device range */
+llend =
+int128_add(int128_makes64(iotlb->addr_mask + 1), int128_makes64(iova));
+if (int128_gt(llend, int128_make64(v->iova_range.last))) {
+error_report("RAM section out of device range (max=0x%" PRIx64
+ ", end addr=0x%" PRIx64 ")",
+ v->iova_range.last, int128_get64(llend));
+return;
+}
+
+vhost_vdpa_iotlb_batch_begin_once(v);
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+
+
+ret =
+vhost_vdpa_dma_map(v, iova, iotlb->addr_mask + 1, vaddr,
read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, iova, iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+Int128 end;
+int iommu_idx;
+IOMMUMemoryRegion *iommu_mr;
+int ret;
+
+iommu_mr = IOMMU_MEMORY_REGION(section->mr);
+
+iommu = g_malloc0(sizeof(*iommu));
+end = int128_add(int128_make64(section->offset_within_region),
+section->size);
+end = int128_sub(end, int128_one());
+iommu_idx = memory_region_iommu_attrs_to_index(iommu_mr,
+MEMTXATTRS_UNSPECIFIED);
+
+iommu->iommu_mr = iommu_mr;
+
+iommu_notifier_init(
+>n, vhost_vdpa_iommu_map_notify, IOMMU_NOTIFIER_IOTLB_EVENTS,
+section->offset_within_region, int128_get64(end), iommu_idx);
+iommu->iommu_offset =
+section->offset_within_address_space - section->offset_within_region;
+iommu->dev = v;
+
+ret = memory_region_register_iommu_notifier(section->mr, >n, NULL);
+if (ret) {
+g_free(iommu);
+return;
+}
+
+QLIST_INSERT_HEAD(>iommu_list, iommu, iommu_next);
+memory_region_iommu_replay(iommu->iommu_mr, >n);
+
+return;
+}
+
+static void vhost_vdpa_iommu_region_del(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+
+QLIST_FOREACH(iommu, >iommu_list, iommu_next)
+{
+if (MEMORY_REGION(iommu->iommu_mr) == section->mr &&
+iommu->n.start == section->offset_within_region) {
+memory_region_unregister_iommu_notifier(section->mr, >n);
+QLIST_REMOVE(iommu, iommu_next)
[PATCH v10 2/2] vhost-vdpa: add support for vIOMMU
Add support for vIOMMU. add the new function to deal with iommu MR.
- during iommu_region_add register a specific IOMMU notifier,
and store all notifiers in a list.
- during iommu_region_del, compare and delete the IOMMU notifier from the list
Verified in vp_vdpa and vdpa_sim_net driver
Signed-off-by: Cindy Lu
---
hw/virtio/vhost-vdpa.c | 122 ++---
include/hw/virtio/vhost-vdpa.h | 10 +++
2 files changed, 121 insertions(+), 11 deletions(-)
diff --git a/hw/virtio/vhost-vdpa.c b/hw/virtio/vhost-vdpa.c
index 3ff9ce3501..e483d6e60b 100644
--- a/hw/virtio/vhost-vdpa.c
+++ b/hw/virtio/vhost-vdpa.c
@@ -26,6 +26,7 @@
#include "cpu.h"
#include "trace.h"
#include "qapi/error.h"
+#include "hw/virtio/virtio-access.h"
/*
* Return one past the end of the end of section. Be careful with uint64_t
@@ -44,7 +45,6 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
uint64_t iova_min,
uint64_t iova_max)
{
-Int128 llend;
if ((!memory_region_is_ram(section->mr) &&
!memory_region_is_iommu(section->mr)) ||
@@ -61,14 +61,6 @@ static bool
vhost_vdpa_listener_skipped_section(MemoryRegionSection *section,
return true;
}
-llend = vhost_vdpa_section_end(section);
-if (int128_gt(llend, int128_make64(iova_max))) {
-error_report("RAM section out of device range (max=0x%" PRIx64
- ", end addr=0x%" PRIx64 ")",
- iova_max, int128_get64(llend));
-return true;
-}
-
return false;
}
@@ -173,6 +165,106 @@ static void vhost_vdpa_listener_commit(MemoryListener
*listener)
v->iotlb_batch_begin_sent = false;
}
+static void vhost_vdpa_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
+{
+struct vdpa_iommu *iommu = container_of(n, struct vdpa_iommu, n);
+
+hwaddr iova = iotlb->iova + iommu->iommu_offset;
+struct vhost_vdpa *v = iommu->dev;
+void *vaddr;
+int ret;
+
+if (iotlb->target_as != _space_memory) {
+error_report("Wrong target AS \"%s\", only system memory is allowed",
+ iotlb->target_as->name ? iotlb->target_as->name : "none");
+return;
+}
+RCU_READ_LOCK_GUARD();
+vhost_vdpa_iotlb_batch_begin_once(v);
+
+if ((iotlb->perm & IOMMU_RW) != IOMMU_NONE) {
+bool read_only;
+
+if (!memory_get_xlat_addr(iotlb, , NULL, _only, NULL)) {
+return;
+}
+ret =
+vhost_vdpa_dma_map(v, iova, iotlb->addr_mask + 1, vaddr,
read_only);
+if (ret) {
+error_report("vhost_vdpa_dma_map(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ", %p) = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, vaddr, ret);
+}
+} else {
+ret = vhost_vdpa_dma_unmap(v, iova, iotlb->addr_mask + 1);
+if (ret) {
+error_report("vhost_vdpa_dma_unmap(%p, 0x%" HWADDR_PRIx ", "
+ "0x%" HWADDR_PRIx ") = %d (%m)",
+ v, iova, iotlb->addr_mask + 1, ret);
+}
+}
+}
+
+static void vhost_vdpa_iommu_region_add(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener);
+
+struct vdpa_iommu *iommu;
+Int128 end;
+int iommu_idx;
+IOMMUMemoryRegion *iommu_mr;
+int ret;
+
+iommu_mr = IOMMU_MEMORY_REGION(section->mr);
+
+iommu = g_malloc0(sizeof(*iommu));
+end = int128_add(int128_make64(section->offset_within_region),
+section->size);
+end = int128_sub(end, int128_one());
+iommu_idx = memory_region_iommu_attrs_to_index(iommu_mr,
+MEMTXATTRS_UNSPECIFIED);
+
+iommu->iommu_mr = iommu_mr;
+
+iommu_notifier_init(
+>n, vhost_vdpa_iommu_map_notify, IOMMU_NOTIFIER_IOTLB_EVENTS,
+section->offset_within_region, int128_get64(end), iommu_idx);
+iommu->iommu_offset =
+section->offset_within_address_space - section->offset_within_region;
+iommu->dev = v;
+
+ret = memory_region_register_iommu_notifier(section->mr, >n, NULL);
+if (ret) {
+g_free(iommu);
+return;
+}
+
+QLIST_INSERT_HEAD(>iommu_list, iommu, iommu_next);
+memory_region_iommu_replay(iommu->iommu_mr, >n);
+
+return;
+}
+
+static void vhost_vdpa_iommu_region_del(MemoryListener *listener,
+MemoryRegionSection *section)
+{
+struct vhost_vdpa *v = container_of(listener, struct vhost_vdpa, listener)
[PATCH v10 1/2] vfio: move implement of vfio_get_xlat_addr() to memory.c
- Move the implement vfio_get_xlat_addr to softmmu/memory.c, and
change the name to memory_get_xlat_addr(). So we can use this
function on other devices, such as vDPA device.
- Add a new function vfio_get_xlat_addr in vfio/common.c, and it will check
whether the memory is backed by a discard manager. then device can
have its own warning.
Signed-off-by: Cindy Lu
---
hw/vfio/common.c | 66 +++
include/exec/memory.h | 4 +++
softmmu/memory.c | 72 +++
3 files changed, 81 insertions(+), 61 deletions(-)
diff --git a/hw/vfio/common.c b/hw/vfio/common.c
index ace9562a9b..6bc02b32c8 100644
--- a/hw/vfio/common.c
+++ b/hw/vfio/common.c
@@ -578,45 +578,11 @@ static bool
vfio_listener_skipped_section(MemoryRegionSection *section)
static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr,
ram_addr_t *ram_addr, bool *read_only)
{
-MemoryRegion *mr;
-hwaddr xlat;
-hwaddr len = iotlb->addr_mask + 1;
-bool writable = iotlb->perm & IOMMU_WO;
-
-/*
- * The IOMMU TLB entry we have just covers translation through
- * this IOMMU to its immediate target. We need to translate
- * it the rest of the way through to memory.
- */
-mr = address_space_translate(_space_memory,
- iotlb->translated_addr,
- , , writable,
- MEMTXATTRS_UNSPECIFIED);
-if (!memory_region_is_ram(mr)) {
-error_report("iommu map to non memory area %"HWADDR_PRIx"",
- xlat);
-return false;
-} else if (memory_region_has_ram_discard_manager(mr)) {
-RamDiscardManager *rdm = memory_region_get_ram_discard_manager(mr);
-MemoryRegionSection tmp = {
-.mr = mr,
-.offset_within_region = xlat,
-.size = int128_make64(len),
-};
-
-/*
- * Malicious VMs can map memory into the IOMMU, which is expected
- * to remain discarded. vfio will pin all pages, populating memory.
- * Disallow that. vmstate priorities make sure any RamDiscardManager
- * were already restored before IOMMUs are restored.
- */
-if (!ram_discard_manager_is_populated(rdm, )) {
-error_report("iommu map to discarded memory (e.g., unplugged via"
- " virtio-mem): %"HWADDR_PRIx"",
- iotlb->translated_addr);
-return false;
-}
+bool ret, mr_has_discard_manager;
+ret = memory_get_xlat_addr(iotlb, vaddr, ram_addr, read_only,
+ _has_discard_manager);
+if (ret && mr_has_discard_manager) {
/*
* Malicious VMs might trigger discarding of IOMMU-mapped memory. The
* pages will remain pinned inside vfio until unmapped, resulting in a
@@ -635,29 +601,7 @@ static bool vfio_get_xlat_addr(IOMMUTLBEntry *iotlb, void
**vaddr,
" intended via an IOMMU. It's possible to mitigate "
" by setting/adjusting RLIMIT_MEMLOCK.");
}
-
-/*
- * Translation truncates length to the IOMMU page size,
- * check that it did not truncate too much.
- */
-if (len & iotlb->addr_mask) {
-error_report("iommu has granularity incompatible with target AS");
-return false;
-}
-
-if (vaddr) {
-*vaddr = memory_region_get_ram_ptr(mr) + xlat;
-}
-
-if (ram_addr) {
-*ram_addr = memory_region_get_ram_addr(mr) + xlat;
-}
-
-if (read_only) {
-*read_only = !writable || mr->readonly;
-}
-
-return true;
+return ret;
}
static void vfio_iommu_map_notify(IOMMUNotifier *n, IOMMUTLBEntry *iotlb)
diff --git a/include/exec/memory.h b/include/exec/memory.h
index bfb1de8eea..d1e79c39dc 100644
--- a/include/exec/memory.h
+++ b/include/exec/memory.h
@@ -713,6 +713,10 @@ void
ram_discard_manager_register_listener(RamDiscardManager *rdm,
void ram_discard_manager_unregister_listener(RamDiscardManager *rdm,
RamDiscardListener *rdl);
+bool memory_get_xlat_addr(IOMMUTLBEntry *iotlb, void **vaddr,
+ ram_addr_t *ram_addr, bool *read_only,
+ bool *mr_has_discard_manager);
+
typedef struct CoalescedMemoryRange CoalescedMemoryRange;
typedef struct MemoryRegionIoeventfd MemoryRegionIoeventfd;
diff --git a/softmmu/memory.c b/softmmu/memory.c
index 7ba2048836..bc0be3f62c 100644
--- a/softmmu/memory.c
+++ b/softmmu/memory.c
@@ -33,6 +33,7 @@
#include "qemu/accel.h"
#include "hw/boards.h"
#include "migration/vmstate.h"
+#include "exec/address-spaces.h"
//#define DEBUG_UNASSIGNED
@@ -2121,6 +2122,77
