Re: [PATCH v3 2/2] spapr: Enable virtio iommu_platform=on by default
On Thu, 12 Mar 2020 15:14:06 +1100
Alexey Kardashevskiy wrote:
>
>
> On 10/03/2020 21:43, Greg Kurz wrote:
> > On Thu, 5 Mar 2020 12:59:03 +0100
> > Greg Kurz wrote:
> >
> >> On Thu, 5 Mar 2020 15:30:09 +1100
> >> David Gibson wrote:
> >>
> >>> Traditionally, virtio devices don't do DMA by the usual path on the
> >>> guest platform. In particular they usually bypass any virtual IOMMU
> >>> the guest has, using hypervisor magic to access untranslated guest
> >>> physical addresses.
> >>>
> >>> There's now the optional iommu_platform flag which can tell virtio
> >>> devices to use the platform's normal DMA path, including any IOMMUs.
> >>> That flag was motiviated for the case of hardware virtio
> >>> implementations, but there are other reasons to want it.
> >>>
> >>> Specifically, the fact that the virtio device doesn't use vIOMMU
> >>> translation means that virtio devices are unsafe to pass to nested
> >>> guests, or to use with VFIO userspace drivers inside the guest. This
> >>> is particularly noticeable on the pseries platform which *always* has
> >>> a guest-visible vIOMMU.
> >>>
> >>> Not using the normal DMA path also causes difficulties for the guest
> >>> side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
> >>> While it's theoretically possible to handle this on the guest side,
> >>> it's really fiddly. Given the other problems with the non-translated
> >>> virtio device, let's just enable vIOMMU translation for virtio devices
> >>> by default in the pseries-5.0 (and later) machine types.
> >>>
> >>> This does mean the new machine type will no longer support guest
> >>> kernels older than 4.8, unless they have support for the virtio
> >>> IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
> >>> do).
> >>>
> >>> Signed-off-by: David Gibson
> >>> ---
> >>
> >> The patch looks good but I'm not sure if we're quite ready to merge
> >> it yet. With this applied, I get zero output on a virtio-serial based
> >> console:
> >>
> >> ie.
> >> -chardev stdio,id=con0 -device virtio-serial -device
> >> virtconsole,chardev=con0
> >>
> >> FYI, virtio-serial is a bit broken for spapr with iommu_platform=off
> >> already:
> >>
> >> (1) pressing a key in the console during SLOF or grub has no effect
> >>
> >> (2) the guest kernel boot stays stuck around quiesce
> >>
> >> These are regressions introduced by this SLOF update:
> >>
> >> a363e9ed8731f45674260932a340a0d81c4b0a6f is the first bad commit
> >> commit a363e9ed8731f45674260932a340a0d81c4b0a6f
> >> Author: Alexey Kardashevskiy
> >> Date: Tue Dec 17 11:31:54 2019 +1100
> >> pseries: Update SLOF firmware image
> >>
> >> A trivial fix was already posted on the SLOF list for (1) :
> >>
> >> https://patchwork.ozlabs.org/patch/1249338/
> >>
> >> (2) is still under investigation but the console is _at least_
> >> functional until the guest OS takes control. This is no longer
> >> the case with this patch.
> >>
> >
> > Some progress was made on the SLOF front:
> >
> > https://patchwork.ozlabs.org/project/slof/list/?series=163314
> >
> > With these series applied to SLOF, I can now boot a fedora31 guest
> > with a virtio-serial console and iommu_platform=on... but now
> > I'm trying out other virtio devices supported by SLOF and I'm
> > running into issues around virtio-pci.disable-legacy as mentioned
> > in some other mail...
> >
> > It seems we may not be ready to merge this series yet.
>
>
> fwiw I sent a pull request:
>
> https://lore.kernel.org/qemu-devel/20200312041010.16229-1-...@ozlabs.ru/T/#u
>
Great ! Thanks mate ! :)
>
>
> >
> >>> hw/ppc/spapr.c | 2 ++
> >>> 1 file changed, 2 insertions(+)
> >>>
> >>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> >>> index 3cfc98ac61..5ef099536e 100644
> >>> --- a/hw/ppc/spapr.c
> >>> +++ b/hw/ppc/spapr.c
> >>> @@ -4575,6 +4575,7 @@ static void
> >>> spapr_machine_latest_class_options(MachineClass *mc)
> >>> */
> >>> static GlobalProperty compat[] = {
> >>> { TYPE_VIRTIO_PCI, "disable-legacy", "on", },
> >>> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
> >>> };
> >>>
> >>> mc->alias = "pseries";
> >>> @@ -4622,6 +4623,7 @@ static void
> >>> spapr_machine_4_2_class_options(MachineClass *mc)
> >>> SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
> >>> static GlobalProperty compat[] = {
> >>> { TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
> >>> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
> >>> };
> >>>
> >>> spapr_machine_5_0_class_options(mc);
> >>
> >
>
Re: [PATCH v3 2/2] spapr: Enable virtio iommu_platform=on by default
On 10/03/2020 21:43, Greg Kurz wrote:
> On Thu, 5 Mar 2020 12:59:03 +0100
> Greg Kurz wrote:
>
>> On Thu, 5 Mar 2020 15:30:09 +1100
>> David Gibson wrote:
>>
>>> Traditionally, virtio devices don't do DMA by the usual path on the
>>> guest platform. In particular they usually bypass any virtual IOMMU
>>> the guest has, using hypervisor magic to access untranslated guest
>>> physical addresses.
>>>
>>> There's now the optional iommu_platform flag which can tell virtio
>>> devices to use the platform's normal DMA path, including any IOMMUs.
>>> That flag was motiviated for the case of hardware virtio
>>> implementations, but there are other reasons to want it.
>>>
>>> Specifically, the fact that the virtio device doesn't use vIOMMU
>>> translation means that virtio devices are unsafe to pass to nested
>>> guests, or to use with VFIO userspace drivers inside the guest. This
>>> is particularly noticeable on the pseries platform which *always* has
>>> a guest-visible vIOMMU.
>>>
>>> Not using the normal DMA path also causes difficulties for the guest
>>> side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
>>> While it's theoretically possible to handle this on the guest side,
>>> it's really fiddly. Given the other problems with the non-translated
>>> virtio device, let's just enable vIOMMU translation for virtio devices
>>> by default in the pseries-5.0 (and later) machine types.
>>>
>>> This does mean the new machine type will no longer support guest
>>> kernels older than 4.8, unless they have support for the virtio
>>> IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
>>> do).
>>>
>>> Signed-off-by: David Gibson
>>> ---
>>
>> The patch looks good but I'm not sure if we're quite ready to merge
>> it yet. With this applied, I get zero output on a virtio-serial based
>> console:
>>
>> ie.
>> -chardev stdio,id=con0 -device virtio-serial -device
>> virtconsole,chardev=con0
>>
>> FYI, virtio-serial is a bit broken for spapr with iommu_platform=off already:
>>
>> (1) pressing a key in the console during SLOF or grub has no effect
>>
>> (2) the guest kernel boot stays stuck around quiesce
>>
>> These are regressions introduced by this SLOF update:
>>
>> a363e9ed8731f45674260932a340a0d81c4b0a6f is the first bad commit
>> commit a363e9ed8731f45674260932a340a0d81c4b0a6f
>> Author: Alexey Kardashevskiy
>> Date: Tue Dec 17 11:31:54 2019 +1100
>> pseries: Update SLOF firmware image
>>
>> A trivial fix was already posted on the SLOF list for (1) :
>>
>> https://patchwork.ozlabs.org/patch/1249338/
>>
>> (2) is still under investigation but the console is _at least_
>> functional until the guest OS takes control. This is no longer
>> the case with this patch.
>>
>
> Some progress was made on the SLOF front:
>
> https://patchwork.ozlabs.org/project/slof/list/?series=163314
>
> With these series applied to SLOF, I can now boot a fedora31 guest
> with a virtio-serial console and iommu_platform=on... but now
> I'm trying out other virtio devices supported by SLOF and I'm
> running into issues around virtio-pci.disable-legacy as mentioned
> in some other mail...
>
> It seems we may not be ready to merge this series yet.
fwiw I sent a pull request:
https://lore.kernel.org/qemu-devel/20200312041010.16229-1-...@ozlabs.ru/T/#u
>
>>> hw/ppc/spapr.c | 2 ++
>>> 1 file changed, 2 insertions(+)
>>>
>>> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
>>> index 3cfc98ac61..5ef099536e 100644
>>> --- a/hw/ppc/spapr.c
>>> +++ b/hw/ppc/spapr.c
>>> @@ -4575,6 +4575,7 @@ static void
>>> spapr_machine_latest_class_options(MachineClass *mc)
>>> */
>>> static GlobalProperty compat[] = {
>>> { TYPE_VIRTIO_PCI, "disable-legacy", "on", },
>>> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
>>> };
>>>
>>> mc->alias = "pseries";
>>> @@ -4622,6 +4623,7 @@ static void
>>> spapr_machine_4_2_class_options(MachineClass *mc)
>>> SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
>>> static GlobalProperty compat[] = {
>>> { TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
>>> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
>>> };
>>>
>>> spapr_machine_5_0_class_options(mc);
>>
>
--
Alexey
Re: [PATCH v3 2/2] spapr: Enable virtio iommu_platform=on by default
On Thu, 5 Mar 2020 12:59:03 +0100
Greg Kurz wrote:
> On Thu, 5 Mar 2020 15:30:09 +1100
> David Gibson wrote:
>
> > Traditionally, virtio devices don't do DMA by the usual path on the
> > guest platform. In particular they usually bypass any virtual IOMMU
> > the guest has, using hypervisor magic to access untranslated guest
> > physical addresses.
> >
> > There's now the optional iommu_platform flag which can tell virtio
> > devices to use the platform's normal DMA path, including any IOMMUs.
> > That flag was motiviated for the case of hardware virtio
> > implementations, but there are other reasons to want it.
> >
> > Specifically, the fact that the virtio device doesn't use vIOMMU
> > translation means that virtio devices are unsafe to pass to nested
> > guests, or to use with VFIO userspace drivers inside the guest. This
> > is particularly noticeable on the pseries platform which *always* has
> > a guest-visible vIOMMU.
> >
> > Not using the normal DMA path also causes difficulties for the guest
> > side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
> > While it's theoretically possible to handle this on the guest side,
> > it's really fiddly. Given the other problems with the non-translated
> > virtio device, let's just enable vIOMMU translation for virtio devices
> > by default in the pseries-5.0 (and later) machine types.
> >
> > This does mean the new machine type will no longer support guest
> > kernels older than 4.8, unless they have support for the virtio
> > IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
> > do).
> >
> > Signed-off-by: David Gibson
> > ---
>
> The patch looks good but I'm not sure if we're quite ready to merge
> it yet. With this applied, I get zero output on a virtio-serial based
> console:
>
> ie.
> -chardev stdio,id=con0 -device virtio-serial -device
> virtconsole,chardev=con0
>
> FYI, virtio-serial is a bit broken for spapr with iommu_platform=off already:
>
> (1) pressing a key in the console during SLOF or grub has no effect
>
> (2) the guest kernel boot stays stuck around quiesce
>
> These are regressions introduced by this SLOF update:
>
> a363e9ed8731f45674260932a340a0d81c4b0a6f is the first bad commit
> commit a363e9ed8731f45674260932a340a0d81c4b0a6f
> Author: Alexey Kardashevskiy
> Date: Tue Dec 17 11:31:54 2019 +1100
> pseries: Update SLOF firmware image
>
> A trivial fix was already posted on the SLOF list for (1) :
>
> https://patchwork.ozlabs.org/patch/1249338/
>
> (2) is still under investigation but the console is _at least_
> functional until the guest OS takes control. This is no longer
> the case with this patch.
>
Some progress was made on the SLOF front:
https://patchwork.ozlabs.org/project/slof/list/?series=163314
With these series applied to SLOF, I can now boot a fedora31 guest
with a virtio-serial console and iommu_platform=on... but now
I'm trying out other virtio devices supported by SLOF and I'm
running into issues around virtio-pci.disable-legacy as mentioned
in some other mail...
It seems we may not be ready to merge this series yet.
> > hw/ppc/spapr.c | 2 ++
> > 1 file changed, 2 insertions(+)
> >
> > diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> > index 3cfc98ac61..5ef099536e 100644
> > --- a/hw/ppc/spapr.c
> > +++ b/hw/ppc/spapr.c
> > @@ -4575,6 +4575,7 @@ static void
> > spapr_machine_latest_class_options(MachineClass *mc)
> > */
> > static GlobalProperty compat[] = {
> > { TYPE_VIRTIO_PCI, "disable-legacy", "on", },
> > +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
> > };
> >
> > mc->alias = "pseries";
> > @@ -4622,6 +4623,7 @@ static void
> > spapr_machine_4_2_class_options(MachineClass *mc)
> > SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
> > static GlobalProperty compat[] = {
> > { TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
> > +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
> > };
> >
> > spapr_machine_5_0_class_options(mc);
>
Re: [PATCH v3 2/2] spapr: Enable virtio iommu_platform=on by default
On Thu, 5 Mar 2020 15:30:09 +1100
David Gibson wrote:
> Traditionally, virtio devices don't do DMA by the usual path on the
> guest platform. In particular they usually bypass any virtual IOMMU
> the guest has, using hypervisor magic to access untranslated guest
> physical addresses.
>
> There's now the optional iommu_platform flag which can tell virtio
> devices to use the platform's normal DMA path, including any IOMMUs.
> That flag was motiviated for the case of hardware virtio
> implementations, but there are other reasons to want it.
>
> Specifically, the fact that the virtio device doesn't use vIOMMU
> translation means that virtio devices are unsafe to pass to nested
> guests, or to use with VFIO userspace drivers inside the guest. This
> is particularly noticeable on the pseries platform which *always* has
> a guest-visible vIOMMU.
>
> Not using the normal DMA path also causes difficulties for the guest
> side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
> While it's theoretically possible to handle this on the guest side,
> it's really fiddly. Given the other problems with the non-translated
> virtio device, let's just enable vIOMMU translation for virtio devices
> by default in the pseries-5.0 (and later) machine types.
>
> This does mean the new machine type will no longer support guest
> kernels older than 4.8, unless they have support for the virtio
> IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
> do).
>
> Signed-off-by: David Gibson
> ---
The patch looks good but I'm not sure if we're quite ready to merge
it yet. With this applied, I get zero output on a virtio-serial based
console:
ie.
-chardev stdio,id=con0 -device virtio-serial -device virtconsole,chardev=con0
FYI, virtio-serial is a bit broken for spapr with iommu_platform=off already:
(1) pressing a key in the console during SLOF or grub has no effect
(2) the guest kernel boot stays stuck around quiesce
These are regressions introduced by this SLOF update:
a363e9ed8731f45674260932a340a0d81c4b0a6f is the first bad commit
commit a363e9ed8731f45674260932a340a0d81c4b0a6f
Author: Alexey Kardashevskiy
Date: Tue Dec 17 11:31:54 2019 +1100
pseries: Update SLOF firmware image
A trivial fix was already posted on the SLOF list for (1) :
https://patchwork.ozlabs.org/patch/1249338/
(2) is still under investigation but the console is _at least_
functional until the guest OS takes control. This is no longer
the case with this patch.
> hw/ppc/spapr.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 3cfc98ac61..5ef099536e 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -4575,6 +4575,7 @@ static void
> spapr_machine_latest_class_options(MachineClass *mc)
> */
> static GlobalProperty compat[] = {
> { TYPE_VIRTIO_PCI, "disable-legacy", "on", },
> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
> };
>
> mc->alias = "pseries";
> @@ -4622,6 +4623,7 @@ static void
> spapr_machine_4_2_class_options(MachineClass *mc)
> SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
> static GlobalProperty compat[] = {
> { TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
> };
>
> spapr_machine_5_0_class_options(mc);
[PATCH v3 2/2] spapr: Enable virtio iommu_platform=on by default
Traditionally, virtio devices don't do DMA by the usual path on the
guest platform. In particular they usually bypass any virtual IOMMU
the guest has, using hypervisor magic to access untranslated guest
physical addresses.
There's now the optional iommu_platform flag which can tell virtio
devices to use the platform's normal DMA path, including any IOMMUs.
That flag was motiviated for the case of hardware virtio
implementations, but there are other reasons to want it.
Specifically, the fact that the virtio device doesn't use vIOMMU
translation means that virtio devices are unsafe to pass to nested
guests, or to use with VFIO userspace drivers inside the guest. This
is particularly noticeable on the pseries platform which *always* has
a guest-visible vIOMMU.
Not using the normal DMA path also causes difficulties for the guest
side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
While it's theoretically possible to handle this on the guest side,
it's really fiddly. Given the other problems with the non-translated
virtio device, let's just enable vIOMMU translation for virtio devices
by default in the pseries-5.0 (and later) machine types.
This does mean the new machine type will no longer support guest
kernels older than 4.8, unless they have support for the virtio
IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
do).
Signed-off-by: David Gibson
---
hw/ppc/spapr.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 3cfc98ac61..5ef099536e 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -4575,6 +4575,7 @@ static void
spapr_machine_latest_class_options(MachineClass *mc)
*/
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "on", },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
};
mc->alias = "pseries";
@@ -4622,6 +4623,7 @@ static void spapr_machine_4_2_class_options(MachineClass
*mc)
SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
};
spapr_machine_5_0_class_options(mc);
--
2.24.1
[PATCH v2 2/2] spapr: Enable virtio iommu_platform=on by default
Traditionally, virtio devices don't do DMA by the usual path on the
guest platform. In particular they usually bypass any virtual IOMMU
the guest has, using hypervisor magic to access untranslated guest
physical addresses.
There's now the optional iommu_platform flag which can tell virtio
devices to use the platform's normal DMA path, including any IOMMUs.
That flag was motiviated for the case of hardware virtio
implementations, but there are other reasons to want it.
Specifically, the fact that the virtio device doesn't use vIOMMU
translation means that virtio devices are unsafe to pass to nested
guests, or to use with VFIO userspace drivers inside the guest. This
is particularly noticeable on the pseries platform which *always* has
a guest-visible vIOMMU.
Not using the normal DMA path also causes difficulties for the guest
side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
While it's theoretically possible to handle this on the guest side,
it's really fiddly. Given the other problems with the non-translated
virtio device, let's just enable vIOMMU translation for virtio devices
by default in the pseries-5.0 (and later) machine types.
This does mean the new machine type will no longer support guest
kernels older than 4.8, unless they have support for the virtio
IOMMU_PLATFORM flag backported (which some distro kernels like RHEL7
do).
Signed-off-by: David Gibson
---
hw/ppc/spapr.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 6e1e467cc6..d4f3dcdda5 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -4573,6 +4573,7 @@ static void spapr_machine_5_0_class_options(MachineClass
*mc)
* default behaviour for virtio */
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "on", },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
};
compat_props_add(mc->compat_props, compat, G_N_ELEMENTS(compat));
@@ -4588,6 +4589,7 @@ static void spapr_machine_4_2_class_options(MachineClass
*mc)
SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
};
spapr_machine_5_0_class_options(mc);
--
2.24.1
Re: [PATCH 2/2] spapr: Enable virtio iommu_platform=on by default
On Fri, Feb 07, 2020 at 03:30:55PM +1100, David Gibson wrote:
> Traditionally, virtio devices don't do DMA by the usual path on the
> guest platform. In particular they usually bypass any virtual IOMMU
> the guest has, using hypervisor magic to access untranslated guest
> physical addresses.
>
> There's now the optional iommu_platform flag which can tell virtio
> devices to use the platform's normal DMA path, including any IOMMUs.
> That flag was motiviated for the case of hardware virtio
> implementations, but there are other reasons to want it.
>
> Specifically, the fact that the virtio device doesn't use vIOMMU
> translation means that virtio devices are unsafe to pass to nested
> guests, or to use with VFIO userspace drivers inside the guest. This
> is particularly noticeable on the pseries platform which *always* has
> a guest-visible vIOMMU.
>
> Not using the normal DMA path also causes difficulties for the guest
> side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
> While it's theoretically possible to handle this on the guest side,
> it's really fiddly. Given the other problems with the non-translated
> virtio device, let's just enable vIOMMU translation for virtio devices
> by default in the pseries-5.0 (and later) machine types.
>
> Signed-off-by: David Gibson
Worth noting that since iommu_platform is mandatory for guests,
this disables support for guests older than Linux 4.8.
> ---
> hw/ppc/spapr.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
> index 216d3b34dc..78e031e80a 100644
> --- a/hw/ppc/spapr.c
> +++ b/hw/ppc/spapr.c
> @@ -4518,6 +4518,7 @@ static void
> spapr_machine_5_0_class_options(MachineClass *mc)
> * default behaviour for virtio */
> static GlobalProperty compat[] = {
> { TYPE_VIRTIO_PCI, "disable-legacy", "on", },
> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
> };
>
> compat_props_add(mc->compat_props, compat, G_N_ELEMENTS(compat));
> @@ -4533,6 +4534,7 @@ static void
> spapr_machine_4_2_class_options(MachineClass *mc)
> SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
> static GlobalProperty compat[] = {
> { TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
> +{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
> };
>
> spapr_machine_5_0_class_options(mc);
> --
> 2.24.1
[PATCH 2/2] spapr: Enable virtio iommu_platform=on by default
Traditionally, virtio devices don't do DMA by the usual path on the
guest platform. In particular they usually bypass any virtual IOMMU
the guest has, using hypervisor magic to access untranslated guest
physical addresses.
There's now the optional iommu_platform flag which can tell virtio
devices to use the platform's normal DMA path, including any IOMMUs.
That flag was motiviated for the case of hardware virtio
implementations, but there are other reasons to want it.
Specifically, the fact that the virtio device doesn't use vIOMMU
translation means that virtio devices are unsafe to pass to nested
guests, or to use with VFIO userspace drivers inside the guest. This
is particularly noticeable on the pseries platform which *always* has
a guest-visible vIOMMU.
Not using the normal DMA path also causes difficulties for the guest
side driver when using the upcoming POWER Secure VMs (a.k.a. PEF).
While it's theoretically possible to handle this on the guest side,
it's really fiddly. Given the other problems with the non-translated
virtio device, let's just enable vIOMMU translation for virtio devices
by default in the pseries-5.0 (and later) machine types.
Signed-off-by: David Gibson
---
hw/ppc/spapr.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/hw/ppc/spapr.c b/hw/ppc/spapr.c
index 216d3b34dc..78e031e80a 100644
--- a/hw/ppc/spapr.c
+++ b/hw/ppc/spapr.c
@@ -4518,6 +4518,7 @@ static void spapr_machine_5_0_class_options(MachineClass
*mc)
* default behaviour for virtio */
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "on", },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "on", },
};
compat_props_add(mc->compat_props, compat, G_N_ELEMENTS(compat));
@@ -4533,6 +4534,7 @@ static void spapr_machine_4_2_class_options(MachineClass
*mc)
SpaprMachineClass *smc = SPAPR_MACHINE_CLASS(mc);
static GlobalProperty compat[] = {
{ TYPE_VIRTIO_PCI, "disable-legacy", "auto" },
+{ TYPE_VIRTIO_DEVICE, "iommu_platform", "off", },
};
spapr_machine_5_0_class_options(mc);
--
2.24.1
Re: virtio,iommu_platform=on
On 14/11/2019 19:42, Michael S. Tsirkin wrote: > On Thu, Nov 14, 2019 at 09:58:47AM +1100, Alexey Kardashevskiy wrote: >> >> >> On 13/11/2019 21:00, Michael S. Tsirkin wrote: >>> On Wed, Nov 13, 2019 at 03:44:28PM +1100, Alexey Kardashevskiy wrote: On 12/11/2019 18:08, Michael S. Tsirkin wrote: > On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: >> Hi! >> >> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing >> problems, one of them is SLOF does SCSI bus scan, then it stops the >> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF >> stopped using the devices) and when this happens, I see unassigned >> memory access (see below) which happens because disabling busmaster >> disables IOMMU and QEMU cannot access the rings to do some shutdown. And >> when this happens, the device does not come back even if SLOF re-enables >> it. > > In fact clearing bus master should disable ring access even > without the IOMMU. > Once you do this you should not wait for rings to be processed, > it is safe to assume they won't be touched again and just > free up any buffers that have not been used. > > Why don't you see this without IOMMU? Because without IOMMU, virtio can always access rings, it does not need bus master address space for that. >>> >>> Right and that's a bug in virtio scsi. E.g. virtio net checks >>> bus mastering before each access. >> >> You have to be specific - virtio scsi in the guest or in QEMU? > > If a device accesses memory with bus master on, it's buggy. > >> >>> Which is all well and good, but we can't just break the world >>> so I guess we first need to fix SLOF, and then add >>> a compat property. And maybe keep it broken for >>> legacy ... >>> > It's a bug I think, probably there to work around buggy guests. > > So pls fix this in SLOF and then hopefully we can drop the > work arounds and have clearing bus master actually block DMA. Laszlo suggested writing 0 to the status but this does not seem helping, with both ioeventfd=true/false. It looks like setting/clearing busmaster bit confused memory region caches in QEMU's virtio. I am confused which direction to keep digging to, any suggestions? Thanks, >>> >>> to clarify you reset after setting bus master? right? >> >> >> I was talking about clearing the bus master, and where I call that >> virtio reset does not matter. Thanks, >> >> > > so > > bus master =0 > reset > bus master =1 > > device does not recover? No, it does not. With my recent changes, it seems that only virtio-block cannot recover; virtio-net and virtio-scsi are fine. Thanks, > > >>> >>> > >> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is >> hardly a right fix. >> >> Is this something expected? Thanks, >> >> >> Here is the exact command line: >> >> /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ >> >> -nodefaults \ >> >> -chardev stdio,id=STDIO0,signal=off,mux=on \ >> >> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ >> >> -mon id=MON0,chardev=STDIO0,mode=readline \ >> >> -nographic \ >> >> -vga none \ >> >> -enable-kvm \ >> -m 2G \ >> >> -device >> virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on >> \ >> -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ >> >> -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ >> >> -snapshot \ >> >> -smp 1 \ >> >> -machine pseries \ >> >> -L /home/aik/t/qemu-ppc64-bios/ \ >> >> -trace events=qemu_trace_events \ >> >> -d guest_errors \ >> >> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ >> >> -mon chardev=SOCKET0,mode=control >> >> >> >> Here is the backtrace: >> >> Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts >> (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ >> aik/p/qemu/memory.c:1275 >> 1275return false; >> #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, >> is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 >> #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 >> , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at >> /home/aik/p/qemu/memory.c:1377 >> #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 >> , addr=0x5802, pval=0x7550d410, op=MO_16, >> attrs=...) at /home/aik/p/qemu/memory.c:1418 >> #3 0x1001cad4 in address_space_lduw_internal_cached_slow >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, >> endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 >> #4
Re: virtio,iommu_platform=on
On Thu, Nov 14, 2019 at 09:58:47AM +1100, Alexey Kardashevskiy wrote: > > > On 13/11/2019 21:00, Michael S. Tsirkin wrote: > > On Wed, Nov 13, 2019 at 03:44:28PM +1100, Alexey Kardashevskiy wrote: > >> > >> > >> On 12/11/2019 18:08, Michael S. Tsirkin wrote: > >>> On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: > Hi! > > I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing > problems, one of them is SLOF does SCSI bus scan, then it stops the > virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF > stopped using the devices) and when this happens, I see unassigned > memory access (see below) which happens because disabling busmaster > disables IOMMU and QEMU cannot access the rings to do some shutdown. And > when this happens, the device does not come back even if SLOF re-enables > it. > >>> > >>> In fact clearing bus master should disable ring access even > >>> without the IOMMU. > >>> Once you do this you should not wait for rings to be processed, > >>> it is safe to assume they won't be touched again and just > >>> free up any buffers that have not been used. > >>> > >>> Why don't you see this without IOMMU? > >> > >> Because without IOMMU, virtio can always access rings, it does not need > >> bus master address space for that. > > > > Right and that's a bug in virtio scsi. E.g. virtio net checks > > bus mastering before each access. > > You have to be specific - virtio scsi in the guest or in QEMU? If a device accesses memory with bus master on, it's buggy. > > > Which is all well and good, but we can't just break the world > > so I guess we first need to fix SLOF, and then add > > a compat property. And maybe keep it broken for > > legacy ... > > > >> > >>> It's a bug I think, probably there to work around buggy guests. > >>> > >>> So pls fix this in SLOF and then hopefully we can drop the > >>> work arounds and have clearing bus master actually block DMA. > >> > >> > >> Laszlo suggested writing 0 to the status but this does not seem helping, > >> with both ioeventfd=true/false. It looks like setting/clearing busmaster > >> bit confused memory region caches in QEMU's virtio. I am confused which > >> direction to keep digging to, any suggestions? Thanks, > >> > > > > to clarify you reset after setting bus master? right? > > > I was talking about clearing the bus master, and where I call that > virtio reset does not matter. Thanks, > > so bus master =0 reset bus master =1 device does not recover? > > > > > >> > >>> > Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is > hardly a right fix. > > Is this something expected? Thanks, > > > Here is the exact command line: > > /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ > > -nodefaults \ > > -chardev stdio,id=STDIO0,signal=off,mux=on \ > > -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ > > -mon id=MON0,chardev=STDIO0,mode=readline \ > > -nographic \ > > -vga none \ > > -enable-kvm \ > -m 2G \ > > -device > virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on > \ > -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ > > -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ > > -snapshot \ > > -smp 1 \ > > -machine pseries \ > > -L /home/aik/t/qemu-ppc64-bios/ \ > > -trace events=qemu_trace_events \ > > -d guest_errors \ > > -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ > > -mon chardev=SOCKET0,mode=control > > > > Here is the backtrace: > > Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts > (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ > aik/p/qemu/memory.c:1275 > 1275return false; > #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, > is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 > #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 > , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at > /home/aik/p/qemu/memory.c:1377 > #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 > , addr=0x5802, pval=0x7550d410, op=MO_16, > attrs=...) at /home/aik/p/qemu/memory.c:1418 > #3 0x1001cad4 in address_space_lduw_internal_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, > endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 > #4 0x1001cc84 in address_space_lduw_le_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > /home/aik/p/qemu/memory_ldst.inc.c:249 > #5 0x1019bd80 in
Re: virtio,iommu_platform=on
On 13/11/2019 21:00, Michael S. Tsirkin wrote: > On Wed, Nov 13, 2019 at 03:44:28PM +1100, Alexey Kardashevskiy wrote: >> >> >> On 12/11/2019 18:08, Michael S. Tsirkin wrote: >>> On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: Hi! I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing problems, one of them is SLOF does SCSI bus scan, then it stops the virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF stopped using the devices) and when this happens, I see unassigned memory access (see below) which happens because disabling busmaster disables IOMMU and QEMU cannot access the rings to do some shutdown. And when this happens, the device does not come back even if SLOF re-enables it. >>> >>> In fact clearing bus master should disable ring access even >>> without the IOMMU. >>> Once you do this you should not wait for rings to be processed, >>> it is safe to assume they won't be touched again and just >>> free up any buffers that have not been used. >>> >>> Why don't you see this without IOMMU? >> >> Because without IOMMU, virtio can always access rings, it does not need >> bus master address space for that. > > Right and that's a bug in virtio scsi. E.g. virtio net checks > bus mastering before each access. You have to be specific - virtio scsi in the guest or in QEMU? > Which is all well and good, but we can't just break the world > so I guess we first need to fix SLOF, and then add > a compat property. And maybe keep it broken for > legacy ... > >> >>> It's a bug I think, probably there to work around buggy guests. >>> >>> So pls fix this in SLOF and then hopefully we can drop the >>> work arounds and have clearing bus master actually block DMA. >> >> >> Laszlo suggested writing 0 to the status but this does not seem helping, >> with both ioeventfd=true/false. It looks like setting/clearing busmaster >> bit confused memory region caches in QEMU's virtio. I am confused which >> direction to keep digging to, any suggestions? Thanks, >> > > to clarify you reset after setting bus master? right? I was talking about clearing the bus master, and where I call that virtio reset does not matter. Thanks, > > >> >>> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is hardly a right fix. Is this something expected? Thanks, Here is the exact command line: /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ -nodefaults \ -chardev stdio,id=STDIO0,signal=off,mux=on \ -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ -mon id=MON0,chardev=STDIO0,mode=readline \ -nographic \ -vga none \ -enable-kvm \ -m 2G \ -device virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on \ -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ -snapshot \ -smp 1 \ -machine pseries \ -L /home/aik/t/qemu-ppc64-bios/ \ -trace events=qemu_trace_events \ -d guest_errors \ -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ -mon chardev=SOCKET0,mode=control Here is the backtrace: Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ aik/p/qemu/memory.c:1275 1275return false; #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1377 #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 , addr=0x5802, pval=0x7550d410, op=MO_16, attrs=...) at /home/aik/p/qemu/memory.c:1418 #3 0x1001cad4 in address_space_lduw_internal_cached_slow (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 #4 0x1001cc84 in address_space_lduw_le_cached_slow (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at /home/aik/p/qemu/memory_ldst.inc.c:249 #5 0x1019bd80 in address_space_lduw_le_cached (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, cache=0x7fff68036fa0, pa=0x2) at /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 #8
Re: virtio,iommu_platform=on
On Wed, Nov 13, 2019 at 03:44:28PM +1100, Alexey Kardashevskiy wrote: > > > On 12/11/2019 18:08, Michael S. Tsirkin wrote: > > On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: > >> Hi! > >> > >> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing > >> problems, one of them is SLOF does SCSI bus scan, then it stops the > >> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF > >> stopped using the devices) and when this happens, I see unassigned > >> memory access (see below) which happens because disabling busmaster > >> disables IOMMU and QEMU cannot access the rings to do some shutdown. And > >> when this happens, the device does not come back even if SLOF re-enables > >> it. > > > > In fact clearing bus master should disable ring access even > > without the IOMMU. > > Once you do this you should not wait for rings to be processed, > > it is safe to assume they won't be touched again and just > > free up any buffers that have not been used. > > > > Why don't you see this without IOMMU? > > Because without IOMMU, virtio can always access rings, it does not need > bus master address space for that. Right and that's a bug in virtio scsi. E.g. virtio net checks bus mastering before each access. Which is all well and good, but we can't just break the world so I guess we first need to fix SLOF, and then add a compat property. And maybe keep it broken for legacy ... > > > It's a bug I think, probably there to work around buggy guests. > > > > So pls fix this in SLOF and then hopefully we can drop the > > work arounds and have clearing bus master actually block DMA. > > > Laszlo suggested writing 0 to the status but this does not seem helping, > with both ioeventfd=true/false. It looks like setting/clearing busmaster > bit confused memory region caches in QEMU's virtio. I am confused which > direction to keep digging to, any suggestions? Thanks, > to clarify you reset after setting bus master? right? > > > > >> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is > >> hardly a right fix. > >> > >> Is this something expected? Thanks, > >> > >> > >> Here is the exact command line: > >> > >> /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ > >> > >> -nodefaults \ > >> > >> -chardev stdio,id=STDIO0,signal=off,mux=on \ > >> > >> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ > >> > >> -mon id=MON0,chardev=STDIO0,mode=readline \ > >> > >> -nographic \ > >> > >> -vga none \ > >> > >> -enable-kvm \ > >> -m 2G \ > >> > >> -device > >> virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on > >> \ > >> -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ > >> > >> -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ > >> > >> -snapshot \ > >> > >> -smp 1 \ > >> > >> -machine pseries \ > >> > >> -L /home/aik/t/qemu-ppc64-bios/ \ > >> > >> -trace events=qemu_trace_events \ > >> > >> -d guest_errors \ > >> > >> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ > >> > >> -mon chardev=SOCKET0,mode=control > >> > >> > >> > >> Here is the backtrace: > >> > >> Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts > >> (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ > >> aik/p/qemu/memory.c:1275 > >> 1275return false; > >> #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, > >> is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 > >> #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 > >> , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at > >> /home/aik/p/qemu/memory.c:1377 > >> #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 > >> , addr=0x5802, pval=0x7550d410, op=MO_16, > >> attrs=...) at /home/aik/p/qemu/memory.c:1418 > >> #3 0x1001cad4 in address_space_lduw_internal_cached_slow > >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, > >> endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 > >> #4 0x1001cc84 in address_space_lduw_le_cached_slow > >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > >> /home/aik/p/qemu/memory_ldst.inc.c:249 > >> #5 0x1019bd80 in address_space_lduw_le_cached > >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > >> /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 > >> #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, > >> addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 > >> #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, > >> cache=0x7fff68036fa0, pa=0x2) at > >> /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 > >> #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at > >> /home/aik/p/qemu/hw/virtio/virtio.c:302 > >> #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at > >> /home/aik/p/qemu/hw/virtio/virtio.c:581 > >> #10 0x1019f838 in
Re: virtio,iommu_platform=on
On 13/11/2019 17:23, Alexey Kardashevskiy wrote: > > > On 13/11/2019 16:54, Michael Roth wrote: >> Quoting Alexey Kardashevskiy (2019-11-11 21:53:49) >>> Hi! >>> >>> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing >>> problems, one of them is SLOF does SCSI bus scan, then it stops the >>> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF >>> stopped using the devices) and when this happens, I see unassigned >>> memory access (see below) which happens because disabling busmaster >>> disables IOMMU and QEMU cannot access the rings to do some shutdown. And >>> when this happens, the device does not come back even if SLOF re-enables it. >>> >>> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is >>> hardly a right fix. >> >> I hit the same issue enabling IOMMU for virtio-blk using this branch: >> >> https://github.com/mdroth/SLOF/commits/virtio-iommu >> >> I just sent a tentative fix for QEMU as: >> >> "virtio-pci: disable vring processing when bus-mastering is disabled" >> >> It's an RFC since piggy-backing off dev->broken seems a bit hacky, but >> it seems to fix the issue at least. btw this fixes my issue with disabling bus master as well. >> >> BTW, the SLOF branch above needs cleanup, but it's booting guests okay >> and I was planning to post this week. Where are you at on yours? Maybe >> we should sync up... > > > Mine is here: github.com:aik/SLOF.git virtio-iommu > > Still have to debug a lot, right now virtio-net does not work :-/ > > Want to take over? :) > > -- Alexey
Re: virtio,iommu_platform=on
On 13/11/2019 16:54, Michael Roth wrote: > Quoting Alexey Kardashevskiy (2019-11-11 21:53:49) >> Hi! >> >> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing >> problems, one of them is SLOF does SCSI bus scan, then it stops the >> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF >> stopped using the devices) and when this happens, I see unassigned >> memory access (see below) which happens because disabling busmaster >> disables IOMMU and QEMU cannot access the rings to do some shutdown. And >> when this happens, the device does not come back even if SLOF re-enables it. >> >> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is >> hardly a right fix. > > I hit the same issue enabling IOMMU for virtio-blk using this branch: > > https://github.com/mdroth/SLOF/commits/virtio-iommu > > I just sent a tentative fix for QEMU as: > > "virtio-pci: disable vring processing when bus-mastering is disabled" > > It's an RFC since piggy-backing off dev->broken seems a bit hacky, but > it seems to fix the issue at least. > > BTW, the SLOF branch above needs cleanup, but it's booting guests okay > and I was planning to post this week. Where are you at on yours? Maybe > we should sync up... Mine is here: github.com:aik/SLOF.git virtio-iommu Still have to debug a lot, right now virtio-net does not work :-/ Want to take over? :) -- Alexey
Re: virtio,iommu_platform=on
Quoting Alexey Kardashevskiy (2019-11-11 21:53:49) > Hi! > > I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing > problems, one of them is SLOF does SCSI bus scan, then it stops the > virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF > stopped using the devices) and when this happens, I see unassigned > memory access (see below) which happens because disabling busmaster > disables IOMMU and QEMU cannot access the rings to do some shutdown. And > when this happens, the device does not come back even if SLOF re-enables it. > > Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is > hardly a right fix. I hit the same issue enabling IOMMU for virtio-blk using this branch: https://github.com/mdroth/SLOF/commits/virtio-iommu I just sent a tentative fix for QEMU as: "virtio-pci: disable vring processing when bus-mastering is disabled" It's an RFC since piggy-backing off dev->broken seems a bit hacky, but it seems to fix the issue at least. BTW, the SLOF branch above needs cleanup, but it's booting guests okay and I was planning to post this week. Where are you at on yours? Maybe we should sync up...
Re: virtio,iommu_platform=on
On 12/11/2019 20:06, Laszlo Ersek wrote: > On 11/12/19 04:53, Alexey Kardashevskiy wrote: >> Hi! >> >> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing >> problems, one of them is SLOF does SCSI bus scan, then it stops the >> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF >> stopped using the devices) and when this happens, I see unassigned >> memory access (see below) which happens because disabling busmaster >> disables IOMMU and QEMU cannot access the rings to do some shutdown. And >> when this happens, the device does not come back even if SLOF re-enables it. >> >> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is >> hardly a right fix. >> >> Is this something expected? Thanks, > > Can you perform a virtio reset (write 0 to the virtio-scsi-pci device's > virtio status register) in SLOF, before clearing PCI_COMMAND? The device stops working in SLOF, even if I do not remove bus master bit ever. Weird... > > Thanks, > Laszlo > > >> >> >> Here is the exact command line: >> >> /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ >> >> -nodefaults \ >> >> -chardev stdio,id=STDIO0,signal=off,mux=on \ >> >> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ >> >> -mon id=MON0,chardev=STDIO0,mode=readline \ >> >> -nographic \ >> >> -vga none \ >> >> -enable-kvm \ >> -m 2G \ >> >> -device >> virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on >> \ >> -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ >> >> -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ >> >> -snapshot \ >> >> -smp 1 \ >> >> -machine pseries \ >> >> -L /home/aik/t/qemu-ppc64-bios/ \ >> >> -trace events=qemu_trace_events \ >> >> -d guest_errors \ >> >> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ >> >> -mon chardev=SOCKET0,mode=control >> >> >> >> Here is the backtrace: >> >> Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts >> (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ >> aik/p/qemu/memory.c:1275 >> 1275return false; >> #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, >> is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 >> #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 >> , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at >> /home/aik/p/qemu/memory.c:1377 >> #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 >> , addr=0x5802, pval=0x7550d410, op=MO_16, >> attrs=...) at /home/aik/p/qemu/memory.c:1418 >> #3 0x1001cad4 in address_space_lduw_internal_cached_slow >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, >> endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 >> #4 0x1001cc84 in address_space_lduw_le_cached_slow >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at >> /home/aik/p/qemu/memory_ldst.inc.c:249 >> #5 0x1019bd80 in address_space_lduw_le_cached >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at >> /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 >> #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, >> addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 >> #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, >> cache=0x7fff68036fa0, pa=0x2) at >> /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 >> #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:302 >> #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:581 >> #10 0x1019f838 in virtio_queue_empty (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:612 >> #11 0x101a8fa8 in virtio_queue_host_notifier_aio_poll >> (opaque=0x118c2798) at /home/aik/p/qemu/hw/virtio/virtio.c:3389 >> #12 0x1092679c in run_poll_handlers_once (ctx=0x11212e40, >> timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:520 >> #13 0x10926aec in try_poll_mode (ctx=0x11212e40, >> timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:607 >> #14 0x10926c2c in aio_poll (ctx=0x11212e40, blocking=0x1) at >> /home/aik/p/qemu/util/aio-posix.c:639 >> #15 0x1091fe0c in aio_wait_bh_oneshot (ctx=0x11212e40, >> cb=0x1016f35c , opaque=0x118b9110) at >> /home/aik/p/qemu/util/aio-wait.c:71 >> #16 0x1016fa60 in virtio_scsi_dataplane_stop (vdev=0x118b9110) >> at /home/aik/p/qemu/hw/scsi/virtio-scsi-dataplane.c:211 >> #17 0x10684740 in virtio_bus_stop_ioeventfd (bus=0x118b9098) at >> /home/aik/p/qemu/hw/virtio/virtio-bus.c:245 >> #18 0x10688290 in virtio_pci_stop_ioeventfd (proxy=0x118b0fa0) >> at /home/aik/p/qemu/hw/virtio/virtio-pci.c:292 >> #19 0x106891e8 in virtio_write_config (pci_dev=0x118b0fa0, >> address=0x4, val=0x100100, len=0x4) at >> /home/aik/p/qemu/hw/virtio/virtio-pci.c:613 >> #20
Re: virtio,iommu_platform=on
On 12/11/2019 18:08, Michael S. Tsirkin wrote: > On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: >> Hi! >> >> I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing >> problems, one of them is SLOF does SCSI bus scan, then it stops the >> virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF >> stopped using the devices) and when this happens, I see unassigned >> memory access (see below) which happens because disabling busmaster >> disables IOMMU and QEMU cannot access the rings to do some shutdown. And >> when this happens, the device does not come back even if SLOF re-enables it. > > In fact clearing bus master should disable ring access even > without the IOMMU. > Once you do this you should not wait for rings to be processed, > it is safe to assume they won't be touched again and just > free up any buffers that have not been used. > > Why don't you see this without IOMMU? Because without IOMMU, virtio can always access rings, it does not need bus master address space for that. > It's a bug I think, probably there to work around buggy guests. > > So pls fix this in SLOF and then hopefully we can drop the > work arounds and have clearing bus master actually block DMA. Laszlo suggested writing 0 to the status but this does not seem helping, with both ioeventfd=true/false. It looks like setting/clearing busmaster bit confused memory region caches in QEMU's virtio. I am confused which direction to keep digging to, any suggestions? Thanks, > >> Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is >> hardly a right fix. >> >> Is this something expected? Thanks, >> >> >> Here is the exact command line: >> >> /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ >> >> -nodefaults \ >> >> -chardev stdio,id=STDIO0,signal=off,mux=on \ >> >> -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ >> >> -mon id=MON0,chardev=STDIO0,mode=readline \ >> >> -nographic \ >> >> -vga none \ >> >> -enable-kvm \ >> -m 2G \ >> >> -device >> virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on >> \ >> -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ >> >> -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ >> >> -snapshot \ >> >> -smp 1 \ >> >> -machine pseries \ >> >> -L /home/aik/t/qemu-ppc64-bios/ \ >> >> -trace events=qemu_trace_events \ >> >> -d guest_errors \ >> >> -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ >> >> -mon chardev=SOCKET0,mode=control >> >> >> >> Here is the backtrace: >> >> Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts >> (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ >> aik/p/qemu/memory.c:1275 >> 1275return false; >> #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, >> is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 >> #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 >> , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at >> /home/aik/p/qemu/memory.c:1377 >> #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 >> , addr=0x5802, pval=0x7550d410, op=MO_16, >> attrs=...) at /home/aik/p/qemu/memory.c:1418 >> #3 0x1001cad4 in address_space_lduw_internal_cached_slow >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, >> endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 >> #4 0x1001cc84 in address_space_lduw_le_cached_slow >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at >> /home/aik/p/qemu/memory_ldst.inc.c:249 >> #5 0x1019bd80 in address_space_lduw_le_cached >> (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at >> /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 >> #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, >> addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 >> #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, >> cache=0x7fff68036fa0, pa=0x2) at >> /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 >> #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:302 >> #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:581 >> #10 0x1019f838 in virtio_queue_empty (vq=0x118c2720) at >> /home/aik/p/qemu/hw/virtio/virtio.c:612 >> #11 0x101a8fa8 in virtio_queue_host_notifier_aio_poll >> (opaque=0x118c2798) at /home/aik/p/qemu/hw/virtio/virtio.c:3389 >> #12 0x1092679c in run_poll_handlers_once (ctx=0x11212e40, >> timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:520 >> #13 0x10926aec in try_poll_mode (ctx=0x11212e40, >> timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:607 >> #14 0x10926c2c in aio_poll (ctx=0x11212e40, blocking=0x1) at >> /home/aik/p/qemu/util/aio-posix.c:639 >> #15 0x1091fe0c in aio_wait_bh_oneshot
Re: virtio,iommu_platform=on
On 11/12/19 04:53, Alexey Kardashevskiy wrote: > Hi! > > I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing > problems, one of them is SLOF does SCSI bus scan, then it stops the > virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF > stopped using the devices) and when this happens, I see unassigned > memory access (see below) which happens because disabling busmaster > disables IOMMU and QEMU cannot access the rings to do some shutdown. And > when this happens, the device does not come back even if SLOF re-enables it. > > Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is > hardly a right fix. > > Is this something expected? Thanks, Can you perform a virtio reset (write 0 to the virtio-scsi-pci device's virtio status register) in SLOF, before clearing PCI_COMMAND? Thanks, Laszlo > > > Here is the exact command line: > > /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ > > -nodefaults \ > > -chardev stdio,id=STDIO0,signal=off,mux=on \ > > -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ > > -mon id=MON0,chardev=STDIO0,mode=readline \ > > -nographic \ > > -vga none \ > > -enable-kvm \ > -m 2G \ > > -device > virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on > \ > -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ > > -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ > > -snapshot \ > > -smp 1 \ > > -machine pseries \ > > -L /home/aik/t/qemu-ppc64-bios/ \ > > -trace events=qemu_trace_events \ > > -d guest_errors \ > > -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ > > -mon chardev=SOCKET0,mode=control > > > > Here is the backtrace: > > Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts > (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ > aik/p/qemu/memory.c:1275 > 1275return false; > #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, > is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 > #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 > , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at > /home/aik/p/qemu/memory.c:1377 > #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 > , addr=0x5802, pval=0x7550d410, op=MO_16, > attrs=...) at /home/aik/p/qemu/memory.c:1418 > #3 0x1001cad4 in address_space_lduw_internal_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, > endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 > #4 0x1001cc84 in address_space_lduw_le_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > /home/aik/p/qemu/memory_ldst.inc.c:249 > #5 0x1019bd80 in address_space_lduw_le_cached > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 > #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, > addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 > #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, > cache=0x7fff68036fa0, pa=0x2) at > /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 > #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:302 > #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:581 > #10 0x1019f838 in virtio_queue_empty (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:612 > #11 0x101a8fa8 in virtio_queue_host_notifier_aio_poll > (opaque=0x118c2798) at /home/aik/p/qemu/hw/virtio/virtio.c:3389 > #12 0x1092679c in run_poll_handlers_once (ctx=0x11212e40, > timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:520 > #13 0x10926aec in try_poll_mode (ctx=0x11212e40, > timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:607 > #14 0x10926c2c in aio_poll (ctx=0x11212e40, blocking=0x1) at > /home/aik/p/qemu/util/aio-posix.c:639 > #15 0x1091fe0c in aio_wait_bh_oneshot (ctx=0x11212e40, > cb=0x1016f35c , opaque=0x118b9110) at > /home/aik/p/qemu/util/aio-wait.c:71 > #16 0x1016fa60 in virtio_scsi_dataplane_stop (vdev=0x118b9110) > at /home/aik/p/qemu/hw/scsi/virtio-scsi-dataplane.c:211 > #17 0x10684740 in virtio_bus_stop_ioeventfd (bus=0x118b9098) at > /home/aik/p/qemu/hw/virtio/virtio-bus.c:245 > #18 0x10688290 in virtio_pci_stop_ioeventfd (proxy=0x118b0fa0) > at /home/aik/p/qemu/hw/virtio/virtio-pci.c:292 > #19 0x106891e8 in virtio_write_config (pci_dev=0x118b0fa0, > address=0x4, val=0x100100, len=0x4) at > /home/aik/p/qemu/hw/virtio/virtio-pci.c:613 > #20 0x105b7228 in pci_host_config_write_common > (pci_dev=0x118b0fa0, addr=0x4, limit=0x100, val=0x100100, len=0x4) at > /home/aik/p/qemu/hw/pci/pci_host.c:81 > #21 0x101f7bdc in finish_write_pci_config (spapr=0x11217200, >
Re: virtio,iommu_platform=on
On Tue, Nov 12, 2019 at 02:53:49PM +1100, Alexey Kardashevskiy wrote: > Hi! > > I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing > problems, one of them is SLOF does SCSI bus scan, then it stops the > virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF > stopped using the devices) and when this happens, I see unassigned > memory access (see below) which happens because disabling busmaster > disables IOMMU and QEMU cannot access the rings to do some shutdown. And > when this happens, the device does not come back even if SLOF re-enables it. In fact clearing bus master should disable ring access even without the IOMMU. Once you do this you should not wait for rings to be processed, it is safe to assume they won't be touched again and just free up any buffers that have not been used. Why don't you see this without IOMMU? It's a bug I think, probably there to work around buggy guests. So pls fix this in SLOF and then hopefully we can drop the work arounds and have clearing bus master actually block DMA. > Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is > hardly a right fix. > > Is this something expected? Thanks, > > > Here is the exact command line: > > /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ > > -nodefaults \ > > -chardev stdio,id=STDIO0,signal=off,mux=on \ > > -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ > > -mon id=MON0,chardev=STDIO0,mode=readline \ > > -nographic \ > > -vga none \ > > -enable-kvm \ > -m 2G \ > > -device > virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on > \ > -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ > > -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ > > -snapshot \ > > -smp 1 \ > > -machine pseries \ > > -L /home/aik/t/qemu-ppc64-bios/ \ > > -trace events=qemu_trace_events \ > > -d guest_errors \ > > -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ > > -mon chardev=SOCKET0,mode=control > > > > Here is the backtrace: > > Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts > (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ > aik/p/qemu/memory.c:1275 > 1275return false; > #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, > is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 > #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 > , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at > /home/aik/p/qemu/memory.c:1377 > #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 > , addr=0x5802, pval=0x7550d410, op=MO_16, > attrs=...) at /home/aik/p/qemu/memory.c:1418 > #3 0x1001cad4 in address_space_lduw_internal_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, > endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 > #4 0x1001cc84 in address_space_lduw_le_cached_slow > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > /home/aik/p/qemu/memory_ldst.inc.c:249 > #5 0x1019bd80 in address_space_lduw_le_cached > (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at > /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 > #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, > addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 > #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, > cache=0x7fff68036fa0, pa=0x2) at > /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 > #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:302 > #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:581 > #10 0x1019f838 in virtio_queue_empty (vq=0x118c2720) at > /home/aik/p/qemu/hw/virtio/virtio.c:612 > #11 0x101a8fa8 in virtio_queue_host_notifier_aio_poll > (opaque=0x118c2798) at /home/aik/p/qemu/hw/virtio/virtio.c:3389 > #12 0x1092679c in run_poll_handlers_once (ctx=0x11212e40, > timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:520 > #13 0x10926aec in try_poll_mode (ctx=0x11212e40, > timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:607 > #14 0x10926c2c in aio_poll (ctx=0x11212e40, blocking=0x1) at > /home/aik/p/qemu/util/aio-posix.c:639 > #15 0x1091fe0c in aio_wait_bh_oneshot (ctx=0x11212e40, > cb=0x1016f35c , opaque=0x118b9110) at > /home/aik/p/qemu/util/aio-wait.c:71 > #16 0x1016fa60 in virtio_scsi_dataplane_stop (vdev=0x118b9110) > at /home/aik/p/qemu/hw/scsi/virtio-scsi-dataplane.c:211 > #17 0x10684740 in virtio_bus_stop_ioeventfd (bus=0x118b9098) at > /home/aik/p/qemu/hw/virtio/virtio-bus.c:245 > #18 0x10688290 in virtio_pci_stop_ioeventfd (proxy=0x118b0fa0) > at /home/aik/p/qemu/hw/virtio/virtio-pci.c:292 > #19 0x106891e8 in virtio_write_config (pci_dev=0x118b0fa0, >
virtio,iommu_platform=on
Hi! I am enabling IOMMU for virtio in the pseries firmware (SLOF) and seeing problems, one of them is SLOF does SCSI bus scan, then it stops the virtio-scsi by clearing MMIO|IO|BUSMASTER from PCI_COMMAND (as SLOF stopped using the devices) and when this happens, I see unassigned memory access (see below) which happens because disabling busmaster disables IOMMU and QEMU cannot access the rings to do some shutdown. And when this happens, the device does not come back even if SLOF re-enables it. Hacking SLOF to not clear BUSMASTER makes virtio-scsi work but it is hardly a right fix. Is this something expected? Thanks, Here is the exact command line: /home/aik/pbuild/qemu-garrison2-ppc64/ppc64-softmmu/qemu-system-ppc64 \ -nodefaults \ -chardev stdio,id=STDIO0,signal=off,mux=on \ -device spapr-vty,id=svty0,reg=0x71000110,chardev=STDIO0 \ -mon id=MON0,chardev=STDIO0,mode=readline \ -nographic \ -vga none \ -enable-kvm \ -m 2G \ -device virtio-scsi-pci,id=vscsi0,iommu_platform=on,disable-modern=off,disable-legacy=on \ -drive id=DRIVE0,if=none,file=img/u1804-64le.qcow2,format=qcow2 \ -device scsi-disk,id=scsi-disk0,drive=DRIVE0 \ -snapshot \ -smp 1 \ -machine pseries \ -L /home/aik/t/qemu-ppc64-bios/ \ -trace events=qemu_trace_events \ -d guest_errors \ -chardev socket,id=SOCKET0,server,nowait,path=qemu.mon.ssh59518 \ -mon chardev=SOCKET0,mode=control Here is the backtrace: Thread 5 "qemu-system-ppc" hit Breakpoint 8, unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/ aik/p/qemu/memory.c:1275 1275return false; #0 unassigned_mem_accepts (opaque=0x0, addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1275 #1 0x100a8ac8 in memory_region_access_valid (mr=0x1105c230 , addr=0x5802, size=0x2, is_write=0x0, attrs=...) at /home/aik/p/qemu/memory.c:1377 #2 0x100a8c88 in memory_region_dispatch_read (mr=0x1105c230 , addr=0x5802, pval=0x7550d410, op=MO_16, attrs=...) at /home/aik/p/qemu/memory.c:1418 #3 0x1001cad4 in address_space_lduw_internal_cached_slow (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0, endian=DEVICE_LITTLE_ENDIAN) at /home/aik/p/qemu/memory_ldst.inc.c:211 #4 0x1001cc84 in address_space_lduw_le_cached_slow (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at /home/aik/p/qemu/memory_ldst.inc.c:249 #5 0x1019bd80 in address_space_lduw_le_cached (cache=0x7fff68036fa0, addr=0x2, attrs=..., result=0x0) at /home/aik/p/qemu/include/exec/memory_ldst_cached.inc.h:56 #6 0x1019c10c in lduw_le_phys_cached (cache=0x7fff68036fa0, addr=0x2) at /home/aik/p/qemu/include/exec/memory_ldst_phys.inc.h:91 #7 0x1019d86c in virtio_lduw_phys_cached (vdev=0x118b9110, cache=0x7fff68036fa0, pa=0x2) at /home/aik/p/qemu/include/hw/virtio/virtio-access.h:166 #8 0x1019e648 in vring_avail_idx (vq=0x118c2720) at /home/aik/p/qemu/hw/virtio/virtio.c:302 #9 0x1019f5bc in virtio_queue_split_empty (vq=0x118c2720) at /home/aik/p/qemu/hw/virtio/virtio.c:581 #10 0x1019f838 in virtio_queue_empty (vq=0x118c2720) at /home/aik/p/qemu/hw/virtio/virtio.c:612 #11 0x101a8fa8 in virtio_queue_host_notifier_aio_poll (opaque=0x118c2798) at /home/aik/p/qemu/hw/virtio/virtio.c:3389 #12 0x1092679c in run_poll_handlers_once (ctx=0x11212e40, timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:520 #13 0x10926aec in try_poll_mode (ctx=0x11212e40, timeout=0x7550d7d8) at /home/aik/p/qemu/util/aio-posix.c:607 #14 0x10926c2c in aio_poll (ctx=0x11212e40, blocking=0x1) at /home/aik/p/qemu/util/aio-posix.c:639 #15 0x1091fe0c in aio_wait_bh_oneshot (ctx=0x11212e40, cb=0x1016f35c , opaque=0x118b9110) at /home/aik/p/qemu/util/aio-wait.c:71 #16 0x1016fa60 in virtio_scsi_dataplane_stop (vdev=0x118b9110) at /home/aik/p/qemu/hw/scsi/virtio-scsi-dataplane.c:211 #17 0x10684740 in virtio_bus_stop_ioeventfd (bus=0x118b9098) at /home/aik/p/qemu/hw/virtio/virtio-bus.c:245 #18 0x10688290 in virtio_pci_stop_ioeventfd (proxy=0x118b0fa0) at /home/aik/p/qemu/hw/virtio/virtio-pci.c:292 #19 0x106891e8 in virtio_write_config (pci_dev=0x118b0fa0, address=0x4, val=0x100100, len=0x4) at /home/aik/p/qemu/hw/virtio/virtio-pci.c:613 #20 0x105b7228 in pci_host_config_write_common (pci_dev=0x118b0fa0, addr=0x4, limit=0x100, val=0x100100, len=0x4) at /home/aik/p/qemu/hw/pci/pci_host.c:81 #21 0x101f7bdc in finish_write_pci_config (spapr=0x11217200, buid=0x8002000, addr=0x4, size=0x4, val=0x100100, rets=0x7e7533e0) at /home/aik/p/qemu/hw/ppc/spapr_pci.c:192 #22 0x101f7cec in rtas_ibm_write_pci_config (cpu=0x11540df0, spapr=0x11217200, token=0x2017, nargs=0x5, args=0x7e7533cc, nret=0x1, rets=0x7e7533e0) at /home/aik/p/qemu/hw/ppc/spapr_pci.c:216 #23 0x101f5860 in spapr_rtas_call (cpu=0x11540df0, spapr=0x11217200, token=0x2017, nargs=0x5, args=0x7e7533cc, nret=0x1,
