Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Wed, Sep 29, 2021 at 7:58 PM Igor Mammedov  wrote:
>
> On Tue, 28 Sep 2021 11:47:26 +0100
> Daniel P. Berrangé  wrote:
>
> > On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:
> > >
> > >
> > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > >
> > > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> > > > >
> > > > >
> > > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > > >
> > > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > > > > This change introduces libvirt xml support for the following two 
> > > > > > > pm options:
> [...]
> > > > > The  switch in libvirt for pcie-root-ports
> > > > > currently does not care whether native or acpi hotplug is used. It 
> > > > > simply
> > > > > turns on the hotplug for that particular port. Whether ACPI or native 
> > > > > is
> > > > > used is controlled by this global flag that Julia has introduced in 
> > > > > 6.1.
>
>
> > > > Right so we have
> > > >
>
> *1*)
> following applies to piix4/q35:
>   * ACPI hotplug when enabled, affects _only_ cold-plugged 'bridges'
> since it requires 'slots' being described in DSDT table which
> in current impl. is static table built at reset time.
>
>  (i.e. built-in or 'bridges' specified on command line,
>   where 'bridges' could be PCI-PCI or PCIe-PCI or root/downstream-ports')
>for anything else ('bridges' added with device_add) native hotplug
>is in use (whether it's SHPC or PCI-E native).
>
> ACPI hotplug wiring is done by calling qbus_set_hotplug_handler()
>  * for root bus piix4_pm_realize()/ich9_pm_init()
>  * for anything else acpi_pcihp_device_plug_cb()
>
>
> > > >   * PIIX4
> > > >
> > > >   - acpi-root-pci-hotplug=bool
> > > >
> > > > Whether hotplug is enabled for the root bridge or not
> > > >
> > > >for pci-root controller
> > > >
> > > >
> > > >   - acpi-pci-hotplug-with-bridge-support=bool
> > > >
> > > > Toggles support for ACPI based hotplug across all bridges.
> > > >   If disabled will there will be no hotplug at all for PIIX4 ?
> > > >   Or does 'shpc' come into play in that scenario ?
>
> 'SHPC' hotplug kicks in if it's enabled. (defaults to 'on' except 2.9 machine 
> type)
>
> on q35/APCI side of things we always advertise -all_ hotplug methods available
>
> build_q35_osc_method()
> /*
>  * Always allow native PME, AER (no dependencies)
>  * Allow SHPC (PCI bridges can have SHPC controller)
>  */
> aml_append(if_ctx, aml_and(a_ctrl, aml_int(0x1F), a_ctrl));
>
> bits 0, 1 are Native PCI-E hotplug and SHPC respectively
>
> for PIIX4 we don't have _OSC so it's up to guest OS to make up
> supported methods.
>
> In order of preference:
>   * Windows supports ACPI hotplug then Native PCI-E (SHPC never worked there)

Hmm. from https://www.mail-archive.com/qemu-devel@nongnu.org/msg746673.html :

Me:
> Stupid question. If both native and acpi is enabled which one does OS chose?
> Does this choice vary between OSes and different flavours of the same OS like
> Windows?

Julia: It will always choose native.

re: SHPC this is the reason I thought SHPC was disabled. In my
experiments, Windows did not seem to care about SHPC.

>   * Linux supports ACPI hotplug, SHPC, Native PCI-E
> (SHPC worked poorly due to need to reserve IO for bridges
>  io reservation hinting (impl. later by Marcel))
>
> > > >PIIX combinations
> > > >
> > > >(1) acpi-root-pci-hotplug=yes
> > > >acpi-pci-hotplug-with-bridge-support=yes
> > > >
> > > >  - All bridges have hotplug
> > > >
> > > >(2) acpi-root-pci-hotplug=yes
> > > >acpi-pci-hotplug-with-bridge-support=no
> > > >
> > > >  - No bridges have hotplug
> > > >
> > > >(3) acpi-root-pci-hotplug=no
> > > >acpi-pci-hotplug-with-bridge-support=yes
> > > >
> > > >  - All bridges except root have hotplug
>
> requested by Promox guys, to battle against Windows 'feature' that
> lets any user to unplug sole NIC using an icon on taskbar.

I implemented that on qemu side :-)

>
> (Laine mentioned we have similar per port control for PCI-E
> ('hotplug' property) that was requested by other users
> probably for the same reason)
>
> so acpi-root-pci-hotplug is similar to pcie-root-port.hotplug
> with a difference that the former applies to whole root bus
> on PIIX4, where the later could be controlled per root port.
>
> > > >(4) acpi-root-pci-hotplug=no
> > > >acpi-pci-hotplug-with-bridge-support=no
> > > >
> > > >  - No bridges have hotplug. Essentially identical to (2)
> > >
> > > no (4) is not identical to (2). In (4) no hotplug is enabled. In (2) pci
> > > root bus still has hotplug enabled.
> >
> > So you're saying that acpi-root-pci-hotplug=yes overrides the
> > global request acpi-pci-hotplug-with-bridge-support=no and
> > turns ACPI hotplug back on for the pcie-root
>
> historically ACPI hotplug on root bus was always supported

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Igor Mammedov]

On Tue, 28 Sep 2021 11:47:26 +0100
Daniel P. Berrangé  wrote:

> On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:
> > 
> > 
> > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> >   
> > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:  
> > > >
> > > >
> > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > >  
> > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:  
> > > > > > This change introduces libvirt xml support for the following two pm 
> > > > > > options:
[...]
> > > > The  switch in libvirt for pcie-root-ports
> > > > currently does not care whether native or acpi hotplug is used. It 
> > > > simply
> > > > turns on the hotplug for that particular port. Whether ACPI or native is
> > > > used is controlled by this global flag that Julia has introduced in 
> > > > 6.1.  


> > > Right so we have
> > >

*1*)
following applies to piix4/q35:
  * ACPI hotplug when enabled, affects _only_ cold-plugged 'bridges'
since it requires 'slots' being described in DSDT table which
in current impl. is static table built at reset time.

 (i.e. built-in or 'bridges' specified on command line,
  where 'bridges' could be PCI-PCI or PCIe-PCI or root/downstream-ports')
   for anything else ('bridges' added with device_add) native hotplug
   is in use (whether it's SHPC or PCI-E native).

ACPI hotplug wiring is done by calling qbus_set_hotplug_handler()
 * for root bus piix4_pm_realize()/ich9_pm_init()
 * for anything else acpi_pcihp_device_plug_cb()


> > >   * PIIX4
> > >
> > >   - acpi-root-pci-hotplug=bool
> > >
> > > Whether hotplug is enabled for the root bridge or not
> > >
> > >for pci-root controller
> > >
> > >
> > >   - acpi-pci-hotplug-with-bridge-support=bool
> > >
> > > Toggles support for ACPI based hotplug across all bridges.
> > >   If disabled will there will be no hotplug at all for PIIX4 ?
> > >   Or does 'shpc' come into play in that scenario ?

'SHPC' hotplug kicks in if it's enabled. (defaults to 'on' except 2.9 machine 
type)

on q35/APCI side of things we always advertise -all_ hotplug methods available

build_q35_osc_method()
/*  
 
 * Always allow native PME, AER (no dependencies)   
 
 * Allow SHPC (PCI bridges can have SHPC controller)
 
 */ 
 
aml_append(if_ctx, aml_and(a_ctrl, aml_int(0x1F), a_ctrl));

bits 0, 1 are Native PCI-E hotplug and SHPC respectively 

for PIIX4 we don't have _OSC so it's up to guest OS to make up
supported methods.

In order of preference:
  * Windows supports ACPI hotplug then Native PCI-E (SHPC never worked there)
  * Linux supports ACPI hotplug, SHPC, Native PCI-E
(SHPC worked poorly due to need to reserve IO for bridges
 io reservation hinting (impl. later by Marcel))

> > >PIIX combinations
> > >
> > >(1) acpi-root-pci-hotplug=yes
> > >acpi-pci-hotplug-with-bridge-support=yes
> > >
> > >  - All bridges have hotplug
> > >
> > >(2) acpi-root-pci-hotplug=yes
> > >acpi-pci-hotplug-with-bridge-support=no
> > >
> > >  - No bridges have hotplug
> > >
> > >(3) acpi-root-pci-hotplug=no
> > >acpi-pci-hotplug-with-bridge-support=yes
> > >
> > >  - All bridges except root have hotplug

requested by Promox guys, to battle against Windows 'feature' that
lets any user to unplug sole NIC using an icon on taskbar.

(Laine mentioned we have similar per port control for PCI-E
('hotplug' property) that was requested by other users
probably for the same reason)

so acpi-root-pci-hotplug is similar to pcie-root-port.hotplug
with a difference that the former applies to whole root bus
on PIIX4, where the later could be controlled per root port.

> > >(4) acpi-root-pci-hotplug=no
> > >acpi-pci-hotplug-with-bridge-support=no
> > >
> > >  - No bridges have hotplug. Essentially identical to (2)  
> > 
> > no (4) is not identical to (2). In (4) no hotplug is enabled. In (2) pci
> > root bus still has hotplug enabled.  
> 
> So you're saying that acpi-root-pci-hotplug=yes overrides the
> global request acpi-pci-hotplug-with-bridge-support=no and
> turns ACPI hotplug back on for the pcie-root

historically ACPI hotplug on root bus was always supported
without any option, i.e. acpi-root-pci-hotplug=yes by default.
acpi-pci-hotplug-with-bridge-support does what its name
claims - i.e. adds hotplug for bridges (at least on PIIX4).

> > >   * Q35

 clarification [*1*] still applies

> > >
> > >
> > >   - acpi-pci-hotplug-with-bridge-support=bool
> > >
> > > Toggles support for ACPI based hotplug. If disabled native
> > >   PCIe hotplug is activated instead
> > >
> > >
> > >   * pcie-root-port
> > >
> > >   - hotplug=bool
> > >
> > > 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Daniel P . Berrangé]

On Wed, Sep 29, 2021 at 02:49:32PM +0200, Igor Mammedov wrote:
> On Tue, 28 Sep 2021 11:59:42 +0100
> Daniel P. Berrangé  wrote:
> 
> > On Tue, Sep 28, 2021 at 11:47:26AM +0100, Daniel P. Berrangé wrote:
> > > On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:  
> > > > 
> > > > 
> > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > >   
> > > > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:  
> > > > > >
> > > > > >
> > > > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > > > >  
> > > > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:  
> > > > > > > > This change introduces libvirt xml support for the following 
> > > > > > > > two pm options:
> > > > > > > >
> > > > > > > > 
> > > > > > > >   
> > > > > > > >   
> > > > > > > >   
> > > > > > >
> > > > > > >  
> > > > > > > > +``acpi-hotplug-bridge``
> > > > > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS 
> > > > > > > > ACPI based hotplug support
> > > > > > > > +   for cold plugged bridges. It is available only for x86 
> > > > > > > > guests, both for q35 and pc
> > > > > > > > +   machine types. For pc machines, the support is available 
> > > > > > > > from `QEMU 2.12`. For q35
> > > > > > > > +   machines, the support is available from `QEMU 6.1`. 
> > > > > > > > Examples of cold plugged bridges
> > > > > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > > > > controller). For q35 machines,
> > > > > > > > +   it includes PCIE root ports (pcie-root-port controller). 
> > > > > > > > This is a global option that
> > > > > > > > +   affects all bridges. No other bridge specific option is 
> > > > > > > > required to be specified.  
> > > > > > >
> > > > > > > Can you confirm my understanding of the situation..
> > > > > > >
> > > > > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > > > > >  
> > > > > >
> > > > > > ACPI is the primary means of enabling hotplug. shpc might also have 
> > > > > > a role
> > > > > > here but I think it is disabled. Igor (cc'd) might throw some 
> > > > > > lights on
> > > > > > how shpc comes to play.  
> > > > >
> > > > > Yes, I think it will be important to understand if 'shpc' becomes 
> > > > > relevant
> > > > > when ACPI hotplug is disabled for PCI
> > > > >  
> > > > > >  
> > > > > > >  - q35 / PCIe topology - hotplug historically used native PCIe 
> > > > > > > hotplug,
> > > > > > >  but in 6.1 switched to ACPI
> > > > > > >  
> > > > > >
> > > > > > Correct.
> > > > > >  
> > > > > > > Given, the name "acpi-hotplug-bridge",  am I right that this 
> > > > > > > option
> > > > > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > > > > approach ?  
> > > > > >
> > > > > > Its complicated.
> > > > > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > > > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.  
> > > > >
> > > > > Oh, I mis-read and didn't realize this was controlling the QEMU
> > > > > "acpi-pci-hotplug-with-bridge-support" configuration.
> > > > >
> > > > > With this in mind I think the naming is somewhat misleading. Setting 
> > > > > it
> > > > > to off would give users the impression that hotplug is disabled, which
> > > > > is not the case for Q35 at least. It is just switching to a different
> > > > > hotplug implementation.
> > > > >
> > > > > At least from Q35 pov, I think it would be better to call it
> > > > >
> > > > > hotplug-mode="acpi|pcie"
> > > > >
> > > > > so it is clear that no matter what value it is set to, hotplug
> > > > > is still available.
> > > > >
> > > > > If we also consider PIIX, then depending on the answer wrt shpc
> > > > > above, we might want one of
> > > > >
> > > > > hotplug-mode="acpi|pcie|none"
> > > > > hotplug-mode="acpi|pcie|shpc"
> > > > >  
> > > > 
> > > > If libvirt does not deal with shpc today I think we should not bother 
> > > > with
> > > > shpc at all. We should simply have a boolean mode appropriately named 
> > > > that
> > > > choses between acpi hotplug vs native.  
> > > 
> > > I want to understand what's possible at the qemu hardware level,
> > > so we don't paint ourselves into a corner.
> > > 
> > > IIUC, with shpc we only have a toggle on "pci-bridge" devices,
> > > and those currently have shpc=true by default. There's no shpc
> > > setting on the pci-root, and theres no global setting.  
> > 
> > Opps, I was mislead. They have shpc=false by default due to machine
> > types >= 2.9 overriding it to false
> 
> 
> If I read it correctly, shcp is on by default
> (modulo 2.9 see 2fa356629ed2)

Sigh, so I was mislead twice ! I should have just tested it for
real, which I have now done below which confirms what you say:

$ echo -e "info qtree\r\nquit\r\n" | qemu-system-x86_64 -monitor stdio -device 
pci-bridge,chassis_nr=4 | grep shpc
shpc = true

$ echo -e "info qtree\r\nquit\r\n" | qemu-system-x86_64 -monitor stdio 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Igor Mammedov]

On Tue, 28 Sep 2021 11:59:42 +0100
Daniel P. Berrangé  wrote:

> On Tue, Sep 28, 2021 at 11:47:26AM +0100, Daniel P. Berrangé wrote:
> > On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:  
> > > 
> > > 
> > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > >   
> > > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:  
> > > > >
> > > > >
> > > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > > >  
> > > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:  
> > > > > > > This change introduces libvirt xml support for the following two 
> > > > > > > pm options:
> > > > > > >
> > > > > > > 
> > > > > > >   
> > > > > > >   
> > > > > > >   
> > > > > >
> > > > > >  
> > > > > > > +``acpi-hotplug-bridge``
> > > > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS 
> > > > > > > ACPI based hotplug support
> > > > > > > +   for cold plugged bridges. It is available only for x86 
> > > > > > > guests, both for q35 and pc
> > > > > > > +   machine types. For pc machines, the support is available from 
> > > > > > > `QEMU 2.12`. For q35
> > > > > > > +   machines, the support is available from `QEMU 6.1`. Examples 
> > > > > > > of cold plugged bridges
> > > > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > > > controller). For q35 machines,
> > > > > > > +   it includes PCIE root ports (pcie-root-port controller). This 
> > > > > > > is a global option that
> > > > > > > +   affects all bridges. No other bridge specific option is 
> > > > > > > required to be specified.  
> > > > > >
> > > > > > Can you confirm my understanding of the situation..
> > > > > >
> > > > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > > > >  
> > > > >
> > > > > ACPI is the primary means of enabling hotplug. shpc might also have a 
> > > > > role
> > > > > here but I think it is disabled. Igor (cc'd) might throw some lights 
> > > > > on
> > > > > how shpc comes to play.  
> > > >
> > > > Yes, I think it will be important to understand if 'shpc' becomes 
> > > > relevant
> > > > when ACPI hotplug is disabled for PCI
> > > >  
> > > > >  
> > > > > >  - q35 / PCIe topology - hotplug historically used native PCIe 
> > > > > > hotplug,
> > > > > >  but in 6.1 switched to ACPI
> > > > > >  
> > > > >
> > > > > Correct.
> > > > >  
> > > > > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > > > approach ?  
> > > > >
> > > > > Its complicated.
> > > > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.  
> > > >
> > > > Oh, I mis-read and didn't realize this was controlling the QEMU
> > > > "acpi-pci-hotplug-with-bridge-support" configuration.
> > > >
> > > > With this in mind I think the naming is somewhat misleading. Setting it
> > > > to off would give users the impression that hotplug is disabled, which
> > > > is not the case for Q35 at least. It is just switching to a different
> > > > hotplug implementation.
> > > >
> > > > At least from Q35 pov, I think it would be better to call it
> > > >
> > > > hotplug-mode="acpi|pcie"
> > > >
> > > > so it is clear that no matter what value it is set to, hotplug
> > > > is still available.
> > > >
> > > > If we also consider PIIX, then depending on the answer wrt shpc
> > > > above, we might want one of
> > > >
> > > > hotplug-mode="acpi|pcie|none"
> > > > hotplug-mode="acpi|pcie|shpc"
> > > >  
> > > 
> > > If libvirt does not deal with shpc today I think we should not bother with
> > > shpc at all. We should simply have a boolean mode appropriately named that
> > > choses between acpi hotplug vs native.  
> > 
> > I want to understand what's possible at the qemu hardware level,
> > so we don't paint ourselves into a corner.
> > 
> > IIUC, with shpc we only have a toggle on "pci-bridge" devices,
> > and those currently have shpc=true by default. There's no shpc
> > setting on the pci-root, and theres no global setting.  
> 
> Opps, I was mislead. They have shpc=false by default due to machine
> types >= 2.9 overriding it to false


If I read it correctly, shcp is on by default
(modulo 2.9 see 2fa356629ed2)

> 
> > Seems to imply that if we have acpi-hotplug disabled for PIIX,
> > then there would be no hotplug on the pci-root, but shpc hotplug
> > would still be available on any pci-bridge devices ?  
> 
> Regards,
> Daniel

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, 28 Sep 2021, Laine Stump wrote:

> On 9/28/21 12:54 PM, Ani Sinha wrote:
> >
> >
> > On Tue, Sep 28, 2021 at 10:17 PM Laine Stump  > > wrote:
> >
> > On 9/28/21 4:44 AM, Daniel P. Berrangé wrote:
> >  > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> >  >> This change introduces libvirt xml support for the following two
> > pm options:
> >  >>
> >  >> 
> >  >>    
> >  >>    
> >  >> 
> >  >
> >  >
> >  >> +``acpi-hotplug-bridge``
> >  >> +   :since:`Since 7.8.0` This option enables or disables BIOS
> > ACPI based hotplug support
> >  >> +   for cold plugged bridges. It is available only for x86
> > guests, both for q35 and pc
> >  >> +   machine types. For pc machines, the support is available
> > from `QEMU 2.12`. For q35
> >  >> +   machines, the support is available from `QEMU 6.1`. Examples
> > of cold plugged bridges
> >  >> +   include PCI-PCI bridges for pc machine types (pci-bridge
> > controller). For q35 machines,
> >  >> +   it includes PCIE root ports (pcie-root-port controller).
> > This is a global option that
> >  >> +   affects all bridges. No other bridge specific option is
> > required to be specified.
> >  >
> >  > Can you confirm my understanding of the situation..
> >  >
> >  >   - i440fx / PCI topology - hotplug always uses ACPI
> >  >
> >  >   - q35 / PCIe topology - hotplug historically used native PCIe
> > hotplug,
> >  >                           but in 6.1 switched to ACPI
> >  >
> >  > Given, the name "acpi-hotplug-bridge",  am I right that this option
> >  > has *no* effect, if the q35 machine is using native PCIe hotplug
> >  > approach ?  IOW, is it a no-op until 6.1 based machine types for
> > q35 ?
> >
> > I *think* that in machinetypes where the default is native-pcie
> > hotplug,
> > setting acpi-hotplug-bridge=on
> >
> > 6.1 not only introduced this option/command line in Qemu but also flipped
> > the switch to make ACPI hotplug default for pcie root ports. So there are
> >  no officially released  version of Qemu where this command line exists and
> > the default is native pcie hotplug.
>
> You're assuming that everyone will use the canonical "q35" machinetype rather
> than a specific versioned machinetype (e.g. "pc-q35-6.0"). For pre-6.1
> machinetypes, the default will still be native-pcie hotplug, even when running
> qemu-6.1+.

Yes you are right. If one uses the 6.1 binary but specifies a compat pre
6.1 machine type then yes, native hotplug would be the default and
acpi-hotplug-bridge will be off. If someone forces it on (not sure why
they would though because they could simply use the 6.1 machine), the acpi
hotplug would be enabled and native hotplug would get disabled on q35.


> >
> >
> > will simultaneously enable ACPI hotplug
> > and disable native-pcie hotplug for all pcie-root-ports and
> > pcie-downstream-ports.
> >
> >
> > This is for 6.1 based machines as well.
> >
> > Similarly on 6.1-based machinetypes, setting
> > acpi-hotplug-bridge=off will disable ACPI hotplug and enable
> > native-pcie
> > hotplug.
> >
> > On 440fx, where the default has always been ACPI (and where SHPC
> > hotplug
> > has been disabled), acpi-hotplug-bridge=on will be a NOP, and
> > acpi-hotplug-bridge=off will completely disable hotplug on any
> > pci-bridge (but *not* on pci-root).
> >
> > As for the acpi-hotplug-root option, that is only valid for 440fx, is
> > "on" by default, and when acpi-hotplug-root=off it will completely
> > disable hotplug to any slot on pci-root.
> >
> > (for completeness - when a pcie-root-port or pcie-downstream-port has
> > , that will disable whatever hotplug mode would
> > have been enabled for the controller - no hotplug at all will be
> > possible on that controller. QEMU also has a "native_hotplug" option
> > (not supported in libvirt) for pcie-root-ports and
> > pcie-downstream-ports
> > which can be used to enable native-pcie hotplug on a specific
> > controller
> > when it had been disabled (in favor of ACPI) with the global
> > acpi-hotplug-bridge ).
> >
>
>

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Laine Stump]

On 9/28/21 12:54 PM, Ani Sinha wrote:



On Tue, Sep 28, 2021 at 10:17 PM Laine Stump > wrote:


On 9/28/21 4:44 AM, Daniel P. Berrangé wrote:
 > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
 >> This change introduces libvirt xml support for the following two
pm options:
 >>
 >> 
 >>    
 >>    
 >> 
 >
 >
 >> +``acpi-hotplug-bridge``
 >> +   :since:`Since 7.8.0` This option enables or disables BIOS
ACPI based hotplug support
 >> +   for cold plugged bridges. It is available only for x86
guests, both for q35 and pc
 >> +   machine types. For pc machines, the support is available
from `QEMU 2.12`. For q35
 >> +   machines, the support is available from `QEMU 6.1`. Examples
of cold plugged bridges
 >> +   include PCI-PCI bridges for pc machine types (pci-bridge
controller). For q35 machines,
 >> +   it includes PCIE root ports (pcie-root-port controller).
This is a global option that
 >> +   affects all bridges. No other bridge specific option is
required to be specified.
 >
 > Can you confirm my understanding of the situation..
 >
 >   - i440fx / PCI topology - hotplug always uses ACPI
 >
 >   - q35 / PCIe topology - hotplug historically used native PCIe
hotplug,
 >                           but in 6.1 switched to ACPI
 >
 > Given, the name "acpi-hotplug-bridge",  am I right that this option
 > has *no* effect, if the q35 machine is using native PCIe hotplug
 > approach ?  IOW, is it a no-op until 6.1 based machine types for
q35 ?

I *think* that in machinetypes where the default is native-pcie
hotplug,
setting acpi-hotplug-bridge=on 



6.1 not only introduced this option/command line in Qemu but also 
flipped the switch to make ACPI hotplug default for pcie root ports. So 
there are  no officially released  version of Qemu where this command 
line exists and the default is native pcie hotplug.


You're assuming that everyone will use the canonical "q35" machinetype 
rather than a specific versioned machinetype (e.g. "pc-q35-6.0"). For 
pre-6.1 machinetypes, the default will still be native-pcie hotplug, 
even when running qemu-6.1+.





will simultaneously enable ACPI hotplug
and disable native-pcie hotplug for all pcie-root-ports and
pcie-downstream-ports.


This is for 6.1 based machines as well.

Similarly on 6.1-based machinetypes, setting
acpi-hotplug-bridge=off will disable ACPI hotplug and enable
native-pcie
hotplug.

On 440fx, where the default has always been ACPI (and where SHPC
hotplug
has been disabled), acpi-hotplug-bridge=on will be a NOP, and
acpi-hotplug-bridge=off will completely disable hotplug on any
pci-bridge (but *not* on pci-root).

As for the acpi-hotplug-root option, that is only valid for 440fx, is
"on" by default, and when acpi-hotplug-root=off it will completely
disable hotplug to any slot on pci-root.

(for completeness - when a pcie-root-port or pcie-downstream-port has
, that will disable whatever hotplug mode would
have been enabled for the controller - no hotplug at all will be
possible on that controller. QEMU also has a "native_hotplug" option
(not supported in libvirt) for pcie-root-ports and
pcie-downstream-ports
which can be used to enable native-pcie hotplug on a specific
controller
when it had been disabled (in favor of ACPI) with the global
acpi-hotplug-bridge ).

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, Sep 28, 2021 at 10:17 PM Laine Stump  wrote:

> On 9/28/21 4:44 AM, Daniel P. Berrangé wrote:
> > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> >> This change introduces libvirt xml support for the following two pm
> options:
> >>
> >> 
> >>
> >>
> >> 
> >
> >
> >> +``acpi-hotplug-bridge``
> >> +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI
> based hotplug support
> >> +   for cold plugged bridges. It is available only for x86 guests, both
> for q35 and pc
> >> +   machine types. For pc machines, the support is available from `QEMU
> 2.12`. For q35
> >> +   machines, the support is available from `QEMU 6.1`. Examples of
> cold plugged bridges
> >> +   include PCI-PCI bridges for pc machine types (pci-bridge
> controller). For q35 machines,
> >> +   it includes PCIE root ports (pcie-root-port controller). This is a
> global option that
> >> +   affects all bridges. No other bridge specific option is required to
> be specified.
> >
> > Can you confirm my understanding of the situation..
> >
> >   - i440fx / PCI topology - hotplug always uses ACPI
> >
> >   - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
> >   but in 6.1 switched to ACPI
> >
> > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > has *no* effect, if the q35 machine is using native PCIe hotplug
> > approach ?  IOW, is it a no-op until 6.1 based machine types for q35 ?
>
> I *think* that in machinetypes where the default is native-pcie hotplug,
> setting acpi-hotplug-bridge=on


6.1 not only introduced this option/command line in Qemu but also flipped
the switch to make ACPI hotplug default for pcie root ports. So there are
 no officially released  version of Qemu where this command line exists and
the default is native pcie hotplug.


will simultaneously enable ACPI hotplug
> and disable native-pcie hotplug for all pcie-root-ports and
> pcie-downstream-ports.


This is for 6.1 based machines as well.

Similarly on 6.1-based machinetypes, setting
> acpi-hotplug-bridge=off will disable ACPI hotplug and enable native-pcie
> hotplug.
>
> On 440fx, where the default has always been ACPI (and where SHPC hotplug
> has been disabled), acpi-hotplug-bridge=on will be a NOP, and
> acpi-hotplug-bridge=off will completely disable hotplug on any
> pci-bridge (but *not* on pci-root).
>
> As for the acpi-hotplug-root option, that is only valid for 440fx, is
> "on" by default, and when acpi-hotplug-root=off it will completely
> disable hotplug to any slot on pci-root.
>
> (for completeness - when a pcie-root-port or pcie-downstream-port has
> , that will disable whatever hotplug mode would
> have been enabled for the controller - no hotplug at all will be
> possible on that controller. QEMU also has a "native_hotplug" option
> (not supported in libvirt) for pcie-root-ports and pcie-downstream-ports
> which can be used to enable native-pcie hotplug on a specific controller
> when it had been disabled (in favor of ACPI) with the global
> acpi-hotplug-bridge ).
>
>

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Laine Stump]

On 9/28/21 4:44 AM, Daniel P. Berrangé wrote:

On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:

This change introduces libvirt xml support for the following two pm options:


   
   





+``acpi-hotplug-bridge``
+   :since:`Since 7.8.0` This option enables or disables BIOS ACPI based 
hotplug support
+   for cold plugged bridges. It is available only for x86 guests, both for q35 
and pc
+   machine types. For pc machines, the support is available from `QEMU 2.12`. 
For q35
+   machines, the support is available from `QEMU 6.1`. Examples of cold 
plugged bridges
+   include PCI-PCI bridges for pc machine types (pci-bridge controller). For 
q35 machines,
+   it includes PCIE root ports (pcie-root-port controller). This is a global 
option that
+   affects all bridges. No other bridge specific option is required to be 
specified.


Can you confirm my understanding of the situation..

  - i440fx / PCI topology - hotplug always uses ACPI

  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
  but in 6.1 switched to ACPI

Given, the name "acpi-hotplug-bridge",  am I right that this option
has *no* effect, if the q35 machine is using native PCIe hotplug
approach ?  IOW, is it a no-op until 6.1 based machine types for q35 ?


I *think* that in machinetypes where the default is native-pcie hotplug, 
setting acpi-hotplug-bridge=on will simultaneously enable ACPI hotplug 
and disable native-pcie hotplug for all pcie-root-ports and 
pcie-downstream-ports. Similarly on 6.1-based machinetypes, setting 
acpi-hotplug-bridge=off will disable ACPI hotplug and enable native-pcie 
hotplug.


On 440fx, where the default has always been ACPI (and where SHPC hotplug 
has been disabled), acpi-hotplug-bridge=on will be a NOP, and 
acpi-hotplug-bridge=off will completely disable hotplug on any 
pci-bridge (but *not* on pci-root).


As for the acpi-hotplug-root option, that is only valid for 440fx, is 
"on" by default, and when acpi-hotplug-root=off it will completely 
disable hotplug to any slot on pci-root.


(for completeness - when a pcie-root-port or pcie-downstream-port has 
, that will disable whatever hotplug mode would 
have been enabled for the controller - no hotplug at all will be 
possible on that controller. QEMU also has a "native_hotplug" option 
(not supported in libvirt) for pcie-root-ports and pcie-downstream-ports 
which can be used to enable native-pcie hotplug on a specific controller 
when it had been disabled (in favor of ACPI) with the global 
acpi-hotplug-bridge ).

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:

> On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:
> >
> >
> > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> >
> > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> > > >
> > > >
> > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > >
> > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > > > This change introduces libvirt xml support for the following two pm 
> > > > > > options:
> > > > > >
> > > > > > 
> > > > > >   
> > > > > >   
> > > > > > 
> > > > >
> > > > >
> > > > > > +``acpi-hotplug-bridge``
> > > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI 
> > > > > > based hotplug support
> > > > > > +   for cold plugged bridges. It is available only for x86 guests, 
> > > > > > both for q35 and pc
> > > > > > +   machine types. For pc machines, the support is available from 
> > > > > > `QEMU 2.12`. For q35
> > > > > > +   machines, the support is available from `QEMU 6.1`. Examples of 
> > > > > > cold plugged bridges
> > > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > > controller). For q35 machines,
> > > > > > +   it includes PCIE root ports (pcie-root-port controller). This 
> > > > > > is a global option that
> > > > > > +   affects all bridges. No other bridge specific option is 
> > > > > > required to be specified.
> > > > >
> > > > > Can you confirm my understanding of the situation..
> > > > >
> > > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > > >
> > > >
> > > > ACPI is the primary means of enabling hotplug. shpc might also have a 
> > > > role
> > > > here but I think it is disabled. Igor (cc'd) might throw some lights on
> > > > how shpc comes to play.
> > >
> > > Yes, I think it will be important to understand if 'shpc' becomes relevant
> > > when ACPI hotplug is disabled for PCI
> > >
> > > >
> > > > >  - q35 / PCIe topology - hotplug historically used native PCIe 
> > > > > hotplug,
> > > > >  but in 6.1 switched to ACPI
> > > > >
> > > >
> > > > Correct.
> > > >
> > > > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > > approach ?
> > > >
> > > > Its complicated.
> > > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.
> > >
> > > Oh, I mis-read and didn't realize this was controlling the QEMU
> > > "acpi-pci-hotplug-with-bridge-support" configuration.
> > >
> > > With this in mind I think the naming is somewhat misleading. Setting it
> > > to off would give users the impression that hotplug is disabled, which
> > > is not the case for Q35 at least. It is just switching to a different
> > > hotplug implementation.
> > >
> > > At least from Q35 pov, I think it would be better to call it
> > >
> > > hotplug-mode="acpi|pcie"
> > >
> > > so it is clear that no matter what value it is set to, hotplug
> > > is still available.
> > >
> > > If we also consider PIIX, then depending on the answer wrt shpc
> > > above, we might want one of
> > >
> > > hotplug-mode="acpi|pcie|none"
> > > hotplug-mode="acpi|pcie|shpc"
> > >
> >
> > If libvirt does not deal with shpc today I think we should not bother with
> > shpc at all. We should simply have a boolean mode appropriately named that
> > choses between acpi hotplug vs native.
>
> I want to understand what's possible at the qemu hardware level,
> so we don't paint ourselves into a corner.
>
> IIUC, with shpc we only have a toggle on "pci-bridge" devices,
> and those currently have shpc=true by default. There's no shpc
> setting on the pci-root, and theres no global setting.
>
> Seems to imply that if we have acpi-hotplug disabled for PIIX,
> then there would be no hotplug on the pci-root, but shpc hotplug
> would still be available on any pci-bridge devices ?
>
> > > > Libvirt does not allow it, but by directly using qemu commandline it is
> > > > possible to enable ACPI hotplug for pcie-root-ports by turning
> > > > "acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. 
> > > > In
> > > > that case, as Julia has mentioned in some other thread, the guest OS
> > > > always choses native over ACPI based hotplug.
> > >
> > > I see that is refering to pie-root-port gaining  native_hotplug=bool in 
> > > 6.1
> > >
> > > You say the guest OS always chooses native over ACPI - is that reqiired
> > > in the spec, or is that just how some common guest OS have choosen to
> > > impl it today ?
> > >
> >
> > I am not sure but I *think* this is guest OS dependent too? Julia to
> > rescue here :-)
> >
> > >
> > > > The  switch in libvirt for pcie-root-ports
> > > > currently does not care whether native or acpi hotplug is used. It 
> > > > simply
> > > > turns on the hotplug for that particular port. Whether ACPI or native is
> > > > used is 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Daniel P . Berrangé]

On Tue, Sep 28, 2021 at 11:47:26AM +0100, Daniel P. Berrangé wrote:
> On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:
> > 
> > 
> > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > 
> > > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> > > >
> > > >
> > > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > > >
> > > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > > > This change introduces libvirt xml support for the following two pm 
> > > > > > options:
> > > > > >
> > > > > > 
> > > > > >   
> > > > > >   
> > > > > > 
> > > > >
> > > > >
> > > > > > +``acpi-hotplug-bridge``
> > > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI 
> > > > > > based hotplug support
> > > > > > +   for cold plugged bridges. It is available only for x86 guests, 
> > > > > > both for q35 and pc
> > > > > > +   machine types. For pc machines, the support is available from 
> > > > > > `QEMU 2.12`. For q35
> > > > > > +   machines, the support is available from `QEMU 6.1`. Examples of 
> > > > > > cold plugged bridges
> > > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > > controller). For q35 machines,
> > > > > > +   it includes PCIE root ports (pcie-root-port controller). This 
> > > > > > is a global option that
> > > > > > +   affects all bridges. No other bridge specific option is 
> > > > > > required to be specified.
> > > > >
> > > > > Can you confirm my understanding of the situation..
> > > > >
> > > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > > >
> > > >
> > > > ACPI is the primary means of enabling hotplug. shpc might also have a 
> > > > role
> > > > here but I think it is disabled. Igor (cc'd) might throw some lights on
> > > > how shpc comes to play.
> > >
> > > Yes, I think it will be important to understand if 'shpc' becomes relevant
> > > when ACPI hotplug is disabled for PCI
> > >
> > > >
> > > > >  - q35 / PCIe topology - hotplug historically used native PCIe 
> > > > > hotplug,
> > > > >  but in 6.1 switched to ACPI
> > > > >
> > > >
> > > > Correct.
> > > >
> > > > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > > approach ?
> > > >
> > > > Its complicated.
> > > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.
> > >
> > > Oh, I mis-read and didn't realize this was controlling the QEMU
> > > "acpi-pci-hotplug-with-bridge-support" configuration.
> > >
> > > With this in mind I think the naming is somewhat misleading. Setting it
> > > to off would give users the impression that hotplug is disabled, which
> > > is not the case for Q35 at least. It is just switching to a different
> > > hotplug implementation.
> > >
> > > At least from Q35 pov, I think it would be better to call it
> > >
> > > hotplug-mode="acpi|pcie"
> > >
> > > so it is clear that no matter what value it is set to, hotplug
> > > is still available.
> > >
> > > If we also consider PIIX, then depending on the answer wrt shpc
> > > above, we might want one of
> > >
> > > hotplug-mode="acpi|pcie|none"
> > > hotplug-mode="acpi|pcie|shpc"
> > >
> > 
> > If libvirt does not deal with shpc today I think we should not bother with
> > shpc at all. We should simply have a boolean mode appropriately named that
> > choses between acpi hotplug vs native.
> 
> I want to understand what's possible at the qemu hardware level,
> so we don't paint ourselves into a corner.
> 
> IIUC, with shpc we only have a toggle on "pci-bridge" devices,
> and those currently have shpc=true by default. There's no shpc
> setting on the pci-root, and theres no global setting.

Opps, I was mislead. They have shpc=false by default due to machine
types >= 2.9 overriding it to false

> Seems to imply that if we have acpi-hotplug disabled for PIIX,
> then there would be no hotplug on the pci-root, but shpc hotplug
> would still be available on any pci-bridge devices ?

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Daniel P . Berrangé]

On Tue, Sep 28, 2021 at 03:28:04PM +0530, Ani Sinha wrote:
> 
> 
> On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> 
> > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> > >
> > >
> > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > >
> > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > > This change introduces libvirt xml support for the following two pm 
> > > > > options:
> > > > >
> > > > > 
> > > > >   
> > > > >   
> > > > > 
> > > >
> > > >
> > > > > +``acpi-hotplug-bridge``
> > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI 
> > > > > based hotplug support
> > > > > +   for cold plugged bridges. It is available only for x86 guests, 
> > > > > both for q35 and pc
> > > > > +   machine types. For pc machines, the support is available from 
> > > > > `QEMU 2.12`. For q35
> > > > > +   machines, the support is available from `QEMU 6.1`. Examples of 
> > > > > cold plugged bridges
> > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > controller). For q35 machines,
> > > > > +   it includes PCIE root ports (pcie-root-port controller). This is 
> > > > > a global option that
> > > > > +   affects all bridges. No other bridge specific option is required 
> > > > > to be specified.
> > > >
> > > > Can you confirm my understanding of the situation..
> > > >
> > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > >
> > >
> > > ACPI is the primary means of enabling hotplug. shpc might also have a role
> > > here but I think it is disabled. Igor (cc'd) might throw some lights on
> > > how shpc comes to play.
> >
> > Yes, I think it will be important to understand if 'shpc' becomes relevant
> > when ACPI hotplug is disabled for PCI
> >
> > >
> > > >  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
> > > >  but in 6.1 switched to ACPI
> > > >
> > >
> > > Correct.
> > >
> > > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > approach ?
> > >
> > > Its complicated.
> > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.
> >
> > Oh, I mis-read and didn't realize this was controlling the QEMU
> > "acpi-pci-hotplug-with-bridge-support" configuration.
> >
> > With this in mind I think the naming is somewhat misleading. Setting it
> > to off would give users the impression that hotplug is disabled, which
> > is not the case for Q35 at least. It is just switching to a different
> > hotplug implementation.
> >
> > At least from Q35 pov, I think it would be better to call it
> >
> > hotplug-mode="acpi|pcie"
> >
> > so it is clear that no matter what value it is set to, hotplug
> > is still available.
> >
> > If we also consider PIIX, then depending on the answer wrt shpc
> > above, we might want one of
> >
> > hotplug-mode="acpi|pcie|none"
> > hotplug-mode="acpi|pcie|shpc"
> >
> 
> If libvirt does not deal with shpc today I think we should not bother with
> shpc at all. We should simply have a boolean mode appropriately named that
> choses between acpi hotplug vs native.

I want to understand what's possible at the qemu hardware level,
so we don't paint ourselves into a corner.

IIUC, with shpc we only have a toggle on "pci-bridge" devices,
and those currently have shpc=true by default. There's no shpc
setting on the pci-root, and theres no global setting.

Seems to imply that if we have acpi-hotplug disabled for PIIX,
then there would be no hotplug on the pci-root, but shpc hotplug
would still be available on any pci-bridge devices ?

> > > Libvirt does not allow it, but by directly using qemu commandline it is
> > > possible to enable ACPI hotplug for pcie-root-ports by turning
> > > "acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. In
> > > that case, as Julia has mentioned in some other thread, the guest OS
> > > always choses native over ACPI based hotplug.
> >
> > I see that is refering to pie-root-port gaining  native_hotplug=bool in 6.1
> >
> > You say the guest OS always chooses native over ACPI - is that reqiired
> > in the spec, or is that just how some common guest OS have choosen to
> > impl it today ?
> >
> 
> I am not sure but I *think* this is guest OS dependent too? Julia to
> rescue here :-)
> 
> >
> > > The  switch in libvirt for pcie-root-ports
> > > currently does not care whether native or acpi hotplug is used. It simply
> > > turns on the hotplug for that particular port. Whether ACPI or native is
> > > used is controlled by this global flag that Julia has introduced in 6.1.
> >
> > Right so we have
> >
> >   * PIIX4
> >
> >   - acpi-root-pci-hotplug=bool
> >
> > Whether hotplug is enabled for the root bridge or not
> >
> >for pci-root controller
> >
> >
> >   - acpi-pci-hotplug-with-bridge-support=bool
> 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, Sep 28, 2021 at 3:28 PM Ani Sinha  wrote:
>
>
>
> On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
>
> > On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> > >
> > >
> > > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> > >
> > > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > > This change introduces libvirt xml support for the following two pm 
> > > > > options:
> > > > >
> > > > > 
> > > > >   
> > > > >   
> > > > > 
> > > >
> > > >
> > > > > +``acpi-hotplug-bridge``
> > > > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI 
> > > > > based hotplug support
> > > > > +   for cold plugged bridges. It is available only for x86 guests, 
> > > > > both for q35 and pc
> > > > > +   machine types. For pc machines, the support is available from 
> > > > > `QEMU 2.12`. For q35
> > > > > +   machines, the support is available from `QEMU 6.1`. Examples of 
> > > > > cold plugged bridges
> > > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > > controller). For q35 machines,
> > > > > +   it includes PCIE root ports (pcie-root-port controller). This is 
> > > > > a global option that
> > > > > +   affects all bridges. No other bridge specific option is required 
> > > > > to be specified.
> > > >
> > > > Can you confirm my understanding of the situation..
> > > >
> > > >  - i440fx / PCI topology - hotplug always uses ACPI
> > > >
> > >
> > > ACPI is the primary means of enabling hotplug. shpc might also have a role
> > > here but I think it is disabled. Igor (cc'd) might throw some lights on
> > > how shpc comes to play.
> >
> > Yes, I think it will be important to understand if 'shpc' becomes relevant
> > when ACPI hotplug is disabled for PCI

https://patchwork.kernel.org/project/qemu-devel/patch/1478099802-14188-1-git-send-email-mar...@redhat.com
has some discussions around shpc.

Also
"PCI devices can be hot-plugged into PCI Express to PCI and PCI-PCI Bridges.
The PCI hot-plug into PCI-PCI bridge is ACPI based, whereas hot-plug into
PCI Express to PCI bridges is SHPC-based. They both can work side by side with
the PCI Express native hot-plug."

https://android.googlesource.com/platform/external/qemu/+/emu-master-dev/docs/pcie.txt

My own experiments on i440fx has shown that Windows does not seem to
care about shpc.
Disabling acpi hotplug on bridges disables hotplug.

> >
> > >
> > > >  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
> > > >  but in 6.1 switched to ACPI
> > > >
> > >
> > > Correct.
> > >
> > > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > > approach ?
> > >
> > > Its complicated.
> > > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.
> >
> > Oh, I mis-read and didn't realize this was controlling the QEMU
> > "acpi-pci-hotplug-with-bridge-support" configuration.
> >
> > With this in mind I think the naming is somewhat misleading. Setting it
> > to off would give users the impression that hotplug is disabled, which
> > is not the case for Q35 at least. It is just switching to a different
> > hotplug implementation.
> >
> > At least from Q35 pov, I think it would be better to call it
> >
> > hotplug-mode="acpi|pcie"
> >
> > so it is clear that no matter what value it is set to, hotplug
> > is still available.
> >
> > If we also consider PIIX, then depending on the answer wrt shpc
> > above, we might want one of
> >
> > hotplug-mode="acpi|pcie|none"
> > hotplug-mode="acpi|pcie|shpc"
> >
>
> If libvirt does not deal with shpc today I think we should not bother with
> shpc at all. We should simply have a boolean mode appropriately named that
> choses between acpi hotplug vs native.
>
> >
> > > Libvirt does not allow it, but by directly using qemu commandline it is
> > > possible to enable ACPI hotplug for pcie-root-ports by turning
> > > "acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. In
> > > that case, as Julia has mentioned in some other thread, the guest OS
> > > always choses native over ACPI based hotplug.
> >
> > I see that is refering to pie-root-port gaining  native_hotplug=bool in 6.1
> >
> > You say the guest OS always chooses native over ACPI - is that reqiired
> > in the spec, or is that just how some common guest OS have choosen to
> > impl it today ?
> >
>
> I am not sure but I *think* this is guest OS dependent too? Julia to
> rescue here :-)
>
> >
> > > The  switch in libvirt for pcie-root-ports
> > > currently does not care whether native or acpi hotplug is used. It simply
> > > turns on the hotplug for that particular port. Whether ACPI or native is
> > > used is controlled by this global flag that Julia has introduced in 6.1.
> >
> > Right so we have
> >
> >   * PIIX4
> >
> >   - acpi-root-pci-hotplug=bool
> >
> > 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:

> On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> >
> >
> > On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> >
> > > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > > This change introduces libvirt xml support for the following two pm 
> > > > options:
> > > >
> > > > 
> > > >   
> > > >   
> > > > 
> > >
> > >
> > > > +``acpi-hotplug-bridge``
> > > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI 
> > > > based hotplug support
> > > > +   for cold plugged bridges. It is available only for x86 guests, both 
> > > > for q35 and pc
> > > > +   machine types. For pc machines, the support is available from `QEMU 
> > > > 2.12`. For q35
> > > > +   machines, the support is available from `QEMU 6.1`. Examples of 
> > > > cold plugged bridges
> > > > +   include PCI-PCI bridges for pc machine types (pci-bridge 
> > > > controller). For q35 machines,
> > > > +   it includes PCIE root ports (pcie-root-port controller). This is a 
> > > > global option that
> > > > +   affects all bridges. No other bridge specific option is required to 
> > > > be specified.
> > >
> > > Can you confirm my understanding of the situation..
> > >
> > >  - i440fx / PCI topology - hotplug always uses ACPI
> > >
> >
> > ACPI is the primary means of enabling hotplug. shpc might also have a role
> > here but I think it is disabled. Igor (cc'd) might throw some lights on
> > how shpc comes to play.
>
> Yes, I think it will be important to understand if 'shpc' becomes relevant
> when ACPI hotplug is disabled for PCI
>
> >
> > >  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
> > >  but in 6.1 switched to ACPI
> > >
> >
> > Correct.
> >
> > > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > > has *no* effect, if the q35 machine is using native PCIe hotplug
> > > approach ?
> >
> > Its complicated.
> > With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> > With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.
>
> Oh, I mis-read and didn't realize this was controlling the QEMU
> "acpi-pci-hotplug-with-bridge-support" configuration.
>
> With this in mind I think the naming is somewhat misleading. Setting it
> to off would give users the impression that hotplug is disabled, which
> is not the case for Q35 at least. It is just switching to a different
> hotplug implementation.
>
> At least from Q35 pov, I think it would be better to call it
>
> hotplug-mode="acpi|pcie"
>
> so it is clear that no matter what value it is set to, hotplug
> is still available.
>
> If we also consider PIIX, then depending on the answer wrt shpc
> above, we might want one of
>
> hotplug-mode="acpi|pcie|none"
> hotplug-mode="acpi|pcie|shpc"
>

If libvirt does not deal with shpc today I think we should not bother with
shpc at all. We should simply have a boolean mode appropriately named that
choses between acpi hotplug vs native.

>
> > Libvirt does not allow it, but by directly using qemu commandline it is
> > possible to enable ACPI hotplug for pcie-root-ports by turning
> > "acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. In
> > that case, as Julia has mentioned in some other thread, the guest OS
> > always choses native over ACPI based hotplug.
>
> I see that is refering to pie-root-port gaining  native_hotplug=bool in 6.1
>
> You say the guest OS always chooses native over ACPI - is that reqiired
> in the spec, or is that just how some common guest OS have choosen to
> impl it today ?
>

I am not sure but I *think* this is guest OS dependent too? Julia to
rescue here :-)

>
> > The  switch in libvirt for pcie-root-ports
> > currently does not care whether native or acpi hotplug is used. It simply
> > turns on the hotplug for that particular port. Whether ACPI or native is
> > used is controlled by this global flag that Julia has introduced in 6.1.
>
> Right so we have
>
>   * PIIX4
>
>   - acpi-root-pci-hotplug=bool
>
> Whether hotplug is enabled for the root bridge or not
>
>for pci-root controller
>
>
>   - acpi-pci-hotplug-with-bridge-support=bool
>
> Toggles support for ACPI based hotplug across all bridges.
>   If disabled will there will be no hotplug at all for PIIX4 ?
>   Or does 'shpc' come into play in that scenario ?
>
>
>PIIX combinations
>
>(1) acpi-root-pci-hotplug=yes
>acpi-pci-hotplug-with-bridge-support=yes
>
>  - All bridges have hotplug
>
>(2) acpi-root-pci-hotplug=yes
>acpi-pci-hotplug-with-bridge-support=no
>
>  - No bridges have hotplug
>
>(3) acpi-root-pci-hotplug=no
>acpi-pci-hotplug-with-bridge-support=yes
>
>  - All bridges except root have hotplug
>
>(4) acpi-root-pci-hotplug=no
>acpi-pci-hotplug-with-bridge-support=no
>
>  - No bridges have hotplug. 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Daniel P . Berrangé]

On Tue, Sep 28, 2021 at 02:35:47PM +0530, Ani Sinha wrote:
> 
> 
> On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:
> 
> > On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > > This change introduces libvirt xml support for the following two pm 
> > > options:
> > >
> > > 
> > >   
> > >   
> > > 
> >
> >
> > > +``acpi-hotplug-bridge``
> > > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI based 
> > > hotplug support
> > > +   for cold plugged bridges. It is available only for x86 guests, both 
> > > for q35 and pc
> > > +   machine types. For pc machines, the support is available from `QEMU 
> > > 2.12`. For q35
> > > +   machines, the support is available from `QEMU 6.1`. Examples of cold 
> > > plugged bridges
> > > +   include PCI-PCI bridges for pc machine types (pci-bridge controller). 
> > > For q35 machines,
> > > +   it includes PCIE root ports (pcie-root-port controller). This is a 
> > > global option that
> > > +   affects all bridges. No other bridge specific option is required to 
> > > be specified.
> >
> > Can you confirm my understanding of the situation..
> >
> >  - i440fx / PCI topology - hotplug always uses ACPI
> >
> 
> ACPI is the primary means of enabling hotplug. shpc might also have a role
> here but I think it is disabled. Igor (cc'd) might throw some lights on
> how shpc comes to play.

Yes, I think it will be important to understand if 'shpc' becomes relevant
when ACPI hotplug is disabled for PCI

> 
> >  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
> >  but in 6.1 switched to ACPI
> >
> 
> Correct.
> 
> > Given, the name "acpi-hotplug-bridge",  am I right that this option
> > has *no* effect, if the q35 machine is using native PCIe hotplug
> > approach ?
> 
> Its complicated.
> With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
> With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.

Oh, I mis-read and didn't realize this was controlling the QEMU
"acpi-pci-hotplug-with-bridge-support" configuration.

With this in mind I think the naming is somewhat misleading. Setting it
to off would give users the impression that hotplug is disabled, which
is not the case for Q35 at least. It is just switching to a different
hotplug implementation.

At least from Q35 pov, I think it would be better to call it

hotplug-mode="acpi|pcie"

so it is clear that no matter what value it is set to, hotplug
is still available.

If we also consider PIIX, then depending on the answer wrt shpc
above, we might want one of

hotplug-mode="acpi|pcie|none"
hotplug-mode="acpi|pcie|shpc"


> Libvirt does not allow it, but by directly using qemu commandline it is
> possible to enable ACPI hotplug for pcie-root-ports by turning
> "acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. In
> that case, as Julia has mentioned in some other thread, the guest OS
> always choses native over ACPI based hotplug.

I see that is refering to pie-root-port gaining  native_hotplug=bool in 6.1

You say the guest OS always chooses native over ACPI - is that reqiired
in the spec, or is that just how some common guest OS have choosen to
impl it today ?


> The  switch in libvirt for pcie-root-ports
> currently does not care whether native or acpi hotplug is used. It simply
> turns on the hotplug for that particular port. Whether ACPI or native is
> used is controlled by this global flag that Julia has introduced in 6.1.

Right so we have

  * PIIX4

  - acpi-root-pci-hotplug=bool

Whether hotplug is enabled for the root bridge or not

   for pci-root controller


  - acpi-pci-hotplug-with-bridge-support=bool

Toggles support for ACPI based hotplug across all bridges.
If disabled will there will be no hotplug at all for PIIX4 ?
Or does 'shpc' come into play in that scenario ?


   PIIX combinations

   (1) acpi-root-pci-hotplug=yes
   acpi-pci-hotplug-with-bridge-support=yes

 - All bridges have hotplug

   (2) acpi-root-pci-hotplug=yes
   acpi-pci-hotplug-with-bridge-support=no

 - No bridges have hotplug

   (3) acpi-root-pci-hotplug=no
   acpi-pci-hotplug-with-bridge-support=yes

 - All bridges except root have hotplug

   (4) acpi-root-pci-hotplug=no
   acpi-pci-hotplug-with-bridge-support=no

 - No bridges have hotplug. Essentially identical to (2)


  * Q35


  - acpi-pci-hotplug-with-bridge-support=bool

Toggles support for ACPI based hotplug. If disabled native
PCIe hotplug is activated instead


  * pcie-root-port

  - hotplug=bool

Toggles whether hotplug is enabled on the port. Affects
either native and ACPI based hotplug, depending on what
acpi-pci-hotplug-with-bridge-support=bool is set to ?

   for pcie-root-port controller

  - native_hotplug=bool

Can be used to also enable native hotplug, 

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

On Tue, 28 Sep 2021, Daniel P. Berrangé wrote:

> On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> > This change introduces libvirt xml support for the following two pm options:
> >
> > 
> >   
> >   
> > 
>
>
> > +``acpi-hotplug-bridge``
> > +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI based 
> > hotplug support
> > +   for cold plugged bridges. It is available only for x86 guests, both for 
> > q35 and pc
> > +   machine types. For pc machines, the support is available from `QEMU 
> > 2.12`. For q35
> > +   machines, the support is available from `QEMU 6.1`. Examples of cold 
> > plugged bridges
> > +   include PCI-PCI bridges for pc machine types (pci-bridge controller). 
> > For q35 machines,
> > +   it includes PCIE root ports (pcie-root-port controller). This is a 
> > global option that
> > +   affects all bridges. No other bridge specific option is required to be 
> > specified.
>
> Can you confirm my understanding of the situation..
>
>  - i440fx / PCI topology - hotplug always uses ACPI
>

ACPI is the primary means of enabling hotplug. shpc might also have a role
here but I think it is disabled. Igor (cc'd) might throw some lights on
how shpc comes to play.

>  - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
>  but in 6.1 switched to ACPI
>

Correct.

> Given, the name "acpi-hotplug-bridge",  am I right that this option
> has *no* effect, if the q35 machine is using native PCIe hotplug
> approach ?

Its complicated.
With "acpi-hotplug-bridge" ON, native hotplug is disabled in qemu.
With "acpi-hotplug-bridge" OFF, native hotplug is enabled in qemu.

Libvirt does not allow it, but by directly using qemu commandline it is
possible to enable ACPI hotplug for pcie-root-ports by turning
"acpi-hotplug-bridge" ON _and_ enable native hotplug at the same time. In
that case, as Julia has mentioned in some other thread, the guest OS
always choses native over ACPI based hotplug.

The  switch in libvirt for pcie-root-ports
currently does not care whether native or acpi hotplug is used. It simply
turns on the hotplug for that particular port. Whether ACPI or native is
used is controlled by this global flag that Julia has introduced in 6.1.

> IOW, is it a no-op until 6.1 based machine types for q35 ?
>

Correct. For q35 this command line did not exist until 6.1.

Re: [PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Daniel P . Berrangé]

On Sun, Sep 12, 2021 at 08:56:29AM +0530, Ani Sinha wrote:
> This change introduces libvirt xml support for the following two pm options:
> 
> 
>   
>   
> 


> +``acpi-hotplug-bridge``
> +   :since:`Since 7.8.0` This option enables or disables BIOS ACPI based 
> hotplug support
> +   for cold plugged bridges. It is available only for x86 guests, both for 
> q35 and pc
> +   machine types. For pc machines, the support is available from `QEMU 
> 2.12`. For q35
> +   machines, the support is available from `QEMU 6.1`. Examples of cold 
> plugged bridges
> +   include PCI-PCI bridges for pc machine types (pci-bridge controller). For 
> q35 machines,
> +   it includes PCIE root ports (pcie-root-port controller). This is a global 
> option that
> +   affects all bridges. No other bridge specific option is required to be 
> specified.

Can you confirm my understanding of the situation..

 - i440fx / PCI topology - hotplug always uses ACPI

 - q35 / PCIe topology - hotplug historically used native PCIe hotplug,
 but in 6.1 switched to ACPI

Given, the name "acpi-hotplug-bridge",  am I right that this option
has *no* effect, if the q35 machine is using native PCIe hotplug
approach ?  IOW, is it a no-op until 6.1 based machine types for q35 ?

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

[PATCH v3 3/5] conf: introduce acpi-hotplug-bridge and acpi-root-hotplug pm options

[Ani Sinha]

This change introduces libvirt xml support for the following two pm options:


  
  


The above two options are only available for qemu driver and that too for x86
guests only. Both of them are global options.

'acpi-hotplug-bridge' option enables or disables ACPI hotplug support for cold
plugged bridges. Examples of cold plugged bridges include PCI-PCI bridge
(pci-bridge controller) for pc machines and pcie-root-port controller for q35
machines. Being global options, no other bridge specific options for pci-bridge
controller or pcie-root-port controllers are required. For pc machine type in
x86, this option is available in qemu for a long time, from version 2.1.
Please see the changes in qemu.git:

9e047b982452c6 ("piix4: add acpi pci hotplug support")
133a2da488062e ("pc: acpi: generate AML only for PCI0 devices if PCI bridge 
hotplug is disabled")

For q35 machine type, this was introduced in qemu 6.1 with the following
changes in qemu.git:

(a) c0e427d6eb5fef ("hw/acpi/ich9: Enable ACPI PCI hot-plug")
(b) 17858a16950860 ("hw/acpi/ich9: Set ACPI PCI hot-plug as default on Q35")

The reasons for enabling ACPI based hotplug for PCIe (q35) based machines (as
opposed to native hotplug) are outlined in (b). It is possible that some users
might still want to use native hotplug on PCIe. Therefore, this config option
enables users to choose either ACPI based hotplug or native hotplug for cold
plugged bridges (for example for pcie root port controller in q35 machines).

'acpi-root-hotplug' option enables or disables ACPI based hotplug for PCI root
bus (pci-root controller). This option is only available for pc machine type.
This additional option enables users to disable hotplug for all devices in the
system without adding an additional PCI-PCI bridge, putting the devices behind
the bridge and using the existing "acpi-hotplug-bridge" option to disable
hotplug on that bridge. This feature was introduced from qemu version 5.2 with
the following change in qemu.git:

3d7e78aaf ("Introduce a new flag for i440fx to disable PCI hotplug on the 
root bus")

The above qemu commit describes some compelling reasons why users might to
disable hotplug on PCI root buses.

This change also adds related unit tests to exercise the new conf options.

Signed-off-by: Ani Sinha 
---
 docs/formatdomain.rst | 36 ---
 docs/schemas/domaincommon.rng | 17 +++
 src/conf/domain_conf.c| 21 -
 src/conf/domain_conf.h|  2 +
 .../pc-i440fx-acpi-hotplug-bridge-disable.xml | 17 +++
 .../pc-i440fx-acpi-root-hotplug-disable.xml   | 17 +++
 .../q35-acpi-hotplug-bridge-disable.xml   | 17 +++
 .../pc-i440fx-acpi-hotplug-bridge-disable.xml | 31 +
 .../pc-i440fx-acpi-root-hotplug-disable.xml   | 31 +
 .../q35-acpi-hotplug-bridge-disable.xml   | 45 +++
 tests/qemuxml2xmltest.c   |  9 
 11 files changed, 236 insertions(+), 7 deletions(-)
 create mode 100644 
tests/qemuxml2argvdata/pc-i440fx-acpi-hotplug-bridge-disable.xml
 create mode 100644 
tests/qemuxml2argvdata/pc-i440fx-acpi-root-hotplug-disable.xml
 create mode 100644 tests/qemuxml2argvdata/q35-acpi-hotplug-bridge-disable.xml
 create mode 100644 
tests/qemuxml2xmloutdata/pc-i440fx-acpi-hotplug-bridge-disable.xml
 create mode 100644 
tests/qemuxml2xmloutdata/pc-i440fx-acpi-root-hotplug-disable.xml
 create mode 100644 tests/qemuxml2xmloutdata/q35-acpi-hotplug-bridge-disable.xml

diff --git a/docs/formatdomain.rst b/docs/formatdomain.rst
index ad3b4ea92c..c8b04c625c 100644
--- a/docs/formatdomain.rst
+++ b/docs/formatdomain.rst
@@ -1793,16 +1793,40 @@ advertisements to the guest OS. (NB: Only qemu driver 
support)

  
  
+ 
+ 

...
 
 ``pm``
-   These elements enable ('yes') or disable ('no') BIOS support for S3
-   (suspend-to-mem) and S4 (suspend-to-disk) ACPI sleep states. If nothing is
-   specified, then the hypervisor will be left with its default value.
-   Note: This setting cannot prevent the guest OS from performing a suspend as
-   the guest OS itself can choose to circumvent the unavailability of the sleep
-   states (e.g. S4 by turning off completely).
+   These elements enable ('yes') or disable ('no') certain BIOS advertisements.
+   If nothing is specified, then the hypervisor will be left with its default 
value.
+   The following BIOS options are available:
+
+``suspend-to-mem``
+   support for S3 (suspend-to-mem) ACPI sleep states.
+``suspend-to-disk``
+   support for S4 (suspend-to-disk) ACPI sleep states.
+
+   Note that for the above two options, the setting cannot prevent the guest OS
+   from performing a suspend as the guest OS itself can choose to circumvent 
the
+   unavailability of the sleep states (e.g. S4 by turning off completely).
+
+``acpi-hotplug-bridge``
+   :since:`Since 7.8.0` This option enables or disables BIOS ACPI based 
hotplug support