Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Nicolin Chen]

On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based extra
> root complexes to be associated with SMMU.
> 
> Although this change does not affect functionality at present, it is
> required when we add support for user-creatable SMMUv3 devices in
> future patches.
> 
> Note: Added a specific check to identify pxb-pcie to avoid matching
> pxb-cxl host bridges, which are also of type PCI_HOST_BRIDGE. This
> restriction can be relaxed once support for CXL devices on arm/virt
> is added and validated with SMMUv3.
> 
> Reviewed-by: Jonathan Cameron 
> Reviewed-by: Eric Auger 
> Tested-by: Nathan Chen 
> Tested-by: Eric Auger  
> Signed-off-by: Shameer Kolothum 

Reviewed-by: Nicolin Chen 

With a small suggestion for clarification.

> +/*
> + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based 
> extra
> + * root complexes to be associated with SMMU.
> + */
> +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> +object_dynamic_cast(OBJECT(pci_bus)->parent, TYPE_PCI_HOST_BRIDGE)) {
> +/*
> + * For pxb-pcie, parent_dev will be set. Make sure it is
> + * pxb-pcie indeed.
> + */

/*
 * While pcie.0 doesn't set the parent_dev, either pxb-pcie or pxb-cxl
 * does. Re-test the type to make sure it is pxb-pcie indeed.
 */

> +if (pci_bus->parent_dev) {
> +if (!object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)) {
> +goto out_err;

Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Nicolin Chen]

On Thu, Jul 10, 2025 at 07:27:10AM +, Shameerali Kolothum Thodi wrote:
> > On Wed, Jul 09, 2025 at 08:08:49AM +, Shameerali Kolothum Thodi
> > wrote:
> > > > On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> > > > > @@ -937,11 +939,32 @@ static void smmu_base_realize(DeviceState
> > > > *dev, Error **errp)
> > > > >   g_free, g_free);
> > > > >  s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);
> > > >
> > > > Although this is not introduced by this patch, is there a
> > > > g_hash_table_remove() somewhere in the code?
> > >
> > > g_hash_table_remove()  is to remove a key/value pair, isn't it?
> > 
> > Yes.
> > 
> > > Or you meant
> > > a corresponding free in case of failure here?
> > 
> > Yes. But I saw the other two g_hash_table_new_full calls were not
> > reverted in the exit path either. Then I saw smmu_base_reset_exit
> > does the clean up of those two but not this smmu_pcibus_by_busptr.
> 
> Ok. I think that is by design. The insert for busptr cache happens during
> early stages of Qemu through get_address_space() callback and
> smmu_base_reset_exit() is called after that, just before the Guest boot.
> So if you clean that cache at that time , you need to handle it differently
> at a later stage. Also I don't think it makes much sense to clear busptr
> before the Guest boot as it is not going to become stale unlike configs
> and iotlb.

Hmm, my main point was there is seemingly no "g_hash_table_remove"
for s->smmu_pcibus_by_busptr throughout the vIOMMU code.

> > > It's a realize() fn and errp is set
> > > if something goes wrong and QEMU will exit. Not sure we need an explicit
> > > free here.
> > >
> > > > > +/*
> > > > > + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie
> > based
> > > > extra
> > > > > + * root complexes to be associated with SMMU.
> > > > > + */
> > > > > +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> > > > > +object_dynamic_cast(OBJECT(pci_bus)->parent,
> > > > TYPE_PCI_HOST_BRIDGE)) {
> > > > > +/*
> > > > > + * For pxb-pcie, parent_dev will be set. Make sure it is
> > > > > + * pxb-pcie indeed.
> > > > > + */
> > > > > +if (pci_bus->parent_dev) {
> > > > > +if (!object_dynamic_cast(OBJECT(pci_bus),
> > TYPE_PXB_PCIE_BUS)) {
> > > >
> > > > The pci_bus_is_express(pci_bus) at the top is equivalent to:
> > > > object_dynamic_cast(OBJECT(pci_bus), TYPE_PCIE_BUS)
> > > > Then here it is doing:
> > > > object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)
> > >
> > > Yes.
> > 
> > Hmm?
> > 
> > We have these two types defined as two different strings, right?
> > 
> > #define TYPE_PCIE_BUS "PCIE"
> > #define TYPE_PXB_PCIE_BUS "pxb-pcie-bus"
> > 
> > So the first test is to make sure pci_bus string is "PCIE",
> > then the second one testing the same pci_bus string will
> > never be true?
> >
> 
> It will be true.
> 
> static const TypeInfo pxb_pcie_bus_info = {
> .name  = TYPE_PXB_PCIE_BUS,
> .parent= TYPE_PCIE_BUS,
> .instance_size = sizeof(PXBBus),
> .class_init= pxb_bus_class_init,
> };
> 
> TYPE_PXB_PCIE_BUS has a parent TYPE_PCIE_BUS. And the function
> object_dynamic_cast() does the magic. It will return non-null for an
> exact object type and also for its parents in the QOM hierarchy.

I see. Thanks for the explain.

> > > > So, this checks the same pci_bus but expects two different types?
> >
> > > In QEMU,  we can have three types of PCIe root complexes to be specified
> > for
> > > virt machine.
> > >
> > > 1. default pcie.0 (TYPE_GPEX_HOST --> TYPE_PCIE_HOST_BRIDGE -->
> > TYPE_PCI_HOST_BRIDGE)
> > > 2. pxb-pcie (TYPE_PXB_HOST  -->TYPE_PCI_HOST_BRIDGE)
> > > 2. pxb-cxl (TYPE_PXB_CXL_HOST  --> TYPE_PCI_HOST_BRIDGE)
> > >
> > > The above first check is to see whether the bus is  PCIE && root bus &&
> > parent
> > > of type TYPE_PCI_HOST_BRIDGE. This will identify all the above three
> > cases.
> > >
> > > Both pxb-pcie and pxb-cxl are special extra root complexes based on PCI
> > > expansion bridges and has a parent_dev set(both has pcie.0 has parent
> > bus).
> > >
> > > Hence we check to see parent_dev is set and make sure it is indeed
> > > TYPE_PXB_PCIE_BUS to avoid attaching to pxb-cxl.
> > 
> > I see. That's clear now. I think it'd help by writing:
> > /*
> >  * While pcie.0 doesn't set the parent_dev, either pxb-pcie
> > or
> >  * pxb-cxl does. Re-test the type to make sure it is pxb-pcie.
> >  */
> 
> I think it is already captured in the comments in this patch.

But I couldn't understand until your further clarification in the
mail :(

Thanks
Nicolin

RE: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Shameerali Kolothum Thodi via]

> -Original Message-
> From: Nicolin Chen 
> Sent: Thursday, July 10, 2025 12:54 AM
> To: Shameerali Kolothum Thodi 
> Cc: qemu-...@nongnu.org; qemu-devel@nongnu.org;
> eric.au...@redhat.com; peter.mayd...@linaro.org; j...@nvidia.com;
> ddut...@redhat.com; berra...@redhat.com; imamm...@redhat.com;
> nath...@nvidia.com; mo...@nvidia.com; smost...@google.com;
> gustavo.rom...@linaro.org; m...@redhat.com;
> marcel.apfelb...@gmail.com; Linuxarm ;
> Wangzhou (B) ; jiangkunkun
> ; Jonathan Cameron
> ; zhangfei....@linaro.org
> Subject: Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has
> PCIe Root Complex association
> 
> On Wed, Jul 09, 2025 at 08:08:49AM +, Shameerali Kolothum Thodi
> wrote:
> > > On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> > > > @@ -937,11 +939,32 @@ static void smmu_base_realize(DeviceState
> > > *dev, Error **errp)
> > > >   g_free, g_free);
> > > >  s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);
> > >
> > > Although this is not introduced by this patch, is there a
> > > g_hash_table_remove() somewhere in the code?
> >
> > g_hash_table_remove()  is to remove a key/value pair, isn't it?
> 
> Yes.
> 
> > Or you meant
> > a corresponding free in case of failure here?
> 
> Yes. But I saw the other two g_hash_table_new_full calls were not
> reverted in the exit path either. Then I saw smmu_base_reset_exit
> does the clean up of those two but not this smmu_pcibus_by_busptr.

Ok. I think that is by design. The insert for busptr cache happens during
early stages of Qemu through get_address_space() callback and
smmu_base_reset_exit() is called after that, just before the Guest boot.
So if you clean that cache at that time , you need to handle it differently
at a later stage. Also I don't think it makes much sense to clear busptr
before the Guest boot as it is not going to become stale unlike configs
and iotlb.

> > It's a realize() fn and errp is set
> > if something goes wrong and QEMU will exit. Not sure we need an explicit
> > free here.
> >
> > > > +/*
> > > > + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie
> based
> > > extra
> > > > + * root complexes to be associated with SMMU.
> > > > + */
> > > > +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> > > > +object_dynamic_cast(OBJECT(pci_bus)->parent,
> > > TYPE_PCI_HOST_BRIDGE)) {
> > > > +/*
> > > > + * For pxb-pcie, parent_dev will be set. Make sure it is
> > > > + * pxb-pcie indeed.
> > > > + */
> > > > +if (pci_bus->parent_dev) {
> > > > +if (!object_dynamic_cast(OBJECT(pci_bus),
> TYPE_PXB_PCIE_BUS)) {
> > >
> > > The pci_bus_is_express(pci_bus) at the top is equivalent to:
> > >   object_dynamic_cast(OBJECT(pci_bus), TYPE_PCIE_BUS)
> > > Then here it is doing:
> > >   object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)
> >
> > Yes.
> 
> Hmm?
> 
> We have these two types defined as two different strings, right?
> 
> #define TYPE_PCIE_BUS "PCIE"
> #define TYPE_PXB_PCIE_BUS "pxb-pcie-bus"
> 
> So the first test is to make sure pci_bus string is "PCIE",
> then the second one testing the same pci_bus string will
> never be true?
>

It will be true.

static const TypeInfo pxb_pcie_bus_info = {
.name  = TYPE_PXB_PCIE_BUS,
.parent= TYPE_PCIE_BUS,
.instance_size = sizeof(PXBBus),
.class_init= pxb_bus_class_init,
};

TYPE_PXB_PCIE_BUS has a parent TYPE_PCIE_BUS. And the function
object_dynamic_cast() does the magic. It will return non-null for an
exact object type and also for its parents in the QOM hierarchy.

> > > So, this checks the same pci_bus but expects two different types?
>
> > In QEMU,  we can have three types of PCIe root complexes to be specified
> for
> > virt machine.
> >
> > 1. default pcie.0 (TYPE_GPEX_HOST --> TYPE_PCIE_HOST_BRIDGE -->
> TYPE_PCI_HOST_BRIDGE)
> > 2. pxb-pcie (TYPE_PXB_HOST  -->TYPE_PCI_HOST_BRIDGE)
> > 2. pxb-cxl (TYPE_PXB_CXL_HOST  --> TYPE_PCI_HOST_BRIDGE)
> >
> > The above first check is to see whether the bus is  PCIE && root bus &&
> parent
> > of type TYPE_PCI_HOST_BRIDGE. This will identify all the above three
> cases.
> >
> > Both pxb-pcie and pxb-cxl are special extra root complexes based on PCI
> > expansion bridges and has a parent_dev set(both has pcie.0 has parent
> bus).
> >
> > Hence we check to see parent_dev is set and make sure it is indeed
> > TYPE_PXB_PCIE_BUS to avoid attaching to pxb-cxl.
> 
> I see. That's clear now. I think it'd help by writing:
>   /*
>* While pcie.0 doesn't set the parent_dev, either pxb-pcie
> or
>* pxb-cxl does. Re-test the type to make sure it is pxb-pcie.
>*/

I think it is already captured in the comments in this patch.

Thanks,
Shameer

Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Nicolin Chen]

On Wed, Jul 09, 2025 at 08:08:49AM +, Shameerali Kolothum Thodi wrote:
> > On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> > > @@ -937,11 +939,32 @@ static void smmu_base_realize(DeviceState
> > *dev, Error **errp)
> > >   g_free, g_free);
> > >  s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);
> > 
> > Although this is not introduced by this patch, is there a
> > g_hash_table_remove() somewhere in the code?
> 
> g_hash_table_remove()  is to remove a key/value pair, isn't it?

Yes.

> Or you meant
> a corresponding free in case of failure here?

Yes. But I saw the other two g_hash_table_new_full calls were not
reverted in the exit path either. Then I saw smmu_base_reset_exit
does the clean up of those two but not this smmu_pcibus_by_busptr.

> It's a realize() fn and errp is set
> if something goes wrong and QEMU will exit. Not sure we need an explicit
> free here.
>  
> > > +/*
> > > + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based
> > extra
> > > + * root complexes to be associated with SMMU.
> > > + */
> > > +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> > > +object_dynamic_cast(OBJECT(pci_bus)->parent,
> > TYPE_PCI_HOST_BRIDGE)) {
> > > +/*
> > > + * For pxb-pcie, parent_dev will be set. Make sure it is
> > > + * pxb-pcie indeed.
> > > + */
> > > +if (pci_bus->parent_dev) {
> > > +if (!object_dynamic_cast(OBJECT(pci_bus), 
> > > TYPE_PXB_PCIE_BUS)) {
> > 
> > The pci_bus_is_express(pci_bus) at the top is equivalent to:
> > object_dynamic_cast(OBJECT(pci_bus), TYPE_PCIE_BUS)
> > Then here it is doing:
> > object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)
> 
> Yes.

Hmm?

We have these two types defined as two different strings, right?

#define TYPE_PCIE_BUS "PCIE"
#define TYPE_PXB_PCIE_BUS "pxb-pcie-bus"

So the first test is to make sure pci_bus string is "PCIE",
then the second one testing the same pci_bus string will
never be true?

> > So, this checks the same pci_bus but expects two different types?
> 
> In QEMU,  we can have three types of PCIe root complexes to be specified for
> virt machine. 
> 
> 1. default pcie.0 (TYPE_GPEX_HOST --> TYPE_PCIE_HOST_BRIDGE --> 
> TYPE_PCI_HOST_BRIDGE)
> 2. pxb-pcie (TYPE_PXB_HOST  -->TYPE_PCI_HOST_BRIDGE)
> 2. pxb-cxl (TYPE_PXB_CXL_HOST  --> TYPE_PCI_HOST_BRIDGE)
> 
> The above first check is to see whether the bus is  PCIE && root bus && 
> parent 
> of type TYPE_PCI_HOST_BRIDGE. This will identify all the above three cases.
> 
> Both pxb-pcie and pxb-cxl are special extra root complexes based on PCI
> expansion bridges and has a parent_dev set(both has pcie.0 has parent bus).
> 
> Hence we check to see parent_dev is set and make sure it is indeed 
> TYPE_PXB_PCIE_BUS to avoid attaching to pxb-cxl. 

I see. That's clear now. I think it'd help by writing:
/*
 * While pcie.0 doesn't set the parent_dev, either pxb-pcie or
 * pxb-cxl does. Re-test the type to make sure it is pxb-pcie.
 */

Thanks
Nicolin

RE: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Shameerali Kolothum Thodi via]

> -Original Message-
> From: Nicolin Chen 
> Sent: Tuesday, July 8, 2025 9:57 PM
> To: Shameerali Kolothum Thodi 
> Cc: qemu-...@nongnu.org; qemu-devel@nongnu.org;
> eric.au...@redhat.com; peter.mayd...@linaro.org; j...@nvidia.com;
> ddut...@redhat.com; berra...@redhat.com; imamm...@redhat.com;
> nath...@nvidia.com; mo...@nvidia.com; smost...@google.com;
> gustavo.rom...@linaro.org; m...@redhat.com;
> marcel.apfelb...@gmail.com; Linuxarm ;
> Wangzhou (B) ; jiangkunkun
> ; Jonathan Cameron
> ; zhangfei....@linaro.org
> Subject: Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has
> PCIe Root Complex association
> 
> On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> > @@ -937,11 +939,32 @@ static void smmu_base_realize(DeviceState
> *dev, Error **errp)
> >   g_free, g_free);
> >  s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);
> 
> Although this is not introduced by this patch, is there a
> g_hash_table_remove() somewhere in the code?

g_hash_table_remove()  is to remove a key/value pair, isn't it? Or you meant
a corresponding free in case of failure here? It's a realize() fn and errp is 
set
if something goes wrong and QEMU will exit. Not sure we need an explicit
free here.
 
> > +/*
> > + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based
> extra
> > + * root complexes to be associated with SMMU.
> > + */
> > +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> > +object_dynamic_cast(OBJECT(pci_bus)->parent,
> TYPE_PCI_HOST_BRIDGE)) {
> > +/*
> > + * For pxb-pcie, parent_dev will be set. Make sure it is
> > + * pxb-pcie indeed.
> > + */
> > +if (pci_bus->parent_dev) {
> > +if (!object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)) {
> 
> The pci_bus_is_express(pci_bus) at the top is equivalent to:
>   object_dynamic_cast(OBJECT(pci_bus), TYPE_PCIE_BUS)
> Then here it is doing:
>   object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)

Yes.

> So, this checks the same pci_bus but expects two different types?

In QEMU,  we can have three types of PCIe root complexes to be specified for
virt machine. 

1. default pcie.0 (TYPE_GPEX_HOST --> TYPE_PCIE_HOST_BRIDGE --> 
TYPE_PCI_HOST_BRIDGE)
2. pxb-pcie (TYPE_PXB_HOST  -->TYPE_PCI_HOST_BRIDGE)
2. pxb-cxl (TYPE_PXB_CXL_HOST  --> TYPE_PCI_HOST_BRIDGE)

The above first check is to see whether the bus is  PCIE && root bus && parent 
of type TYPE_PCI_HOST_BRIDGE. This will identify all the above three cases.

Both pxb-pcie and pxb-cxl are special extra root complexes based on PCI
expansion bridges and has a parent_dev set(both has pcie.0 has parent bus).

Hence we check to see parent_dev is set and make sure it is indeed 
TYPE_PXB_PCIE_BUS to avoid attaching to pxb-cxl. 

As mentioned in the commit log above, cxl support for virt is currently in
progress and once it has verified the functionality with SMMUv3
we can relax that check.

> I don't see the code check "PCIe Root Complex" explicitly, which
> should be TYPE_GPEX_HOST?

Hope it is clear now.

Thanks,
Shameer

Re: [PATCH v7 02/12] hw/arm/smmu-common: Check SMMU has PCIe Root Complex association

[Nicolin Chen]

On Tue, Jul 08, 2025 at 04:40:45PM +0100, Shameer Kolothum wrote:
> @@ -937,11 +939,32 @@ static void smmu_base_realize(DeviceState *dev, Error 
> **errp)
>   g_free, g_free);
>  s->smmu_pcibus_by_busptr = g_hash_table_new(NULL, NULL);

Although this is not introduced by this patch, is there a
g_hash_table_remove() somewhere in the code?

> +/*
> + * We only allow default PCIe Root Complex(pcie.0) or pxb-pcie based 
> extra
> + * root complexes to be associated with SMMU.
> + */
> +if (pci_bus_is_express(pci_bus) && pci_bus_is_root(pci_bus) &&
> +object_dynamic_cast(OBJECT(pci_bus)->parent, TYPE_PCI_HOST_BRIDGE)) {
> +/*
> + * For pxb-pcie, parent_dev will be set. Make sure it is
> + * pxb-pcie indeed.
> + */
> +if (pci_bus->parent_dev) {
> +if (!object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)) {

The pci_bus_is_express(pci_bus) at the top is equivalent to:
object_dynamic_cast(OBJECT(pci_bus), TYPE_PCIE_BUS)
Then here it is doing:
object_dynamic_cast(OBJECT(pci_bus), TYPE_PXB_PCIE_BUS)

So, this checks the same pci_bus but expects two different types?

I don't see the code check "PCIe Root Complex" explicitly, which
should be TYPE_GPEX_HOST?

Thanks
Nicolin