Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/27/2016 06:44 PM, Laszlo Ersek wrote:
On 10/27/16 13:27, Marcel Apfelbaum wrote:
On 10/14/2016 02:36 PM, Laszlo Ersek wrote:
On 10/13/16 16:05, Marcel Apfelbaum wrote:
On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote:
+ -device
pci-bridge,id=pci_bridge1,bus=dmi_pci_bridge1[,chassis_nr=x][,addr=y]
\
+ -device ,bus=pci_bridge1[,addr=x]
{13} It would be nice to spell out the valid device addresses (y and x)
here too -- can we use 0 for them? SHPC again?
Can we / should we simply go with >=1 device addresses?
For pci-bridges only - yes. A better idea (I think) is to disable SHPC
by default
from the next QEMU version. Make this slot usable. Sounds OK?
Sure.
+Don't use PCI Express Switches if you don't have too, they use
+an extra PCI bus that may handy to plug another device id it
comes to it.
+
{25} I'd put it as: Don't use PCI Express Switches if you don't have
too, each one of those uses an extra PCI bus (for its Upstream Port)
that could be put to better use with another Root Port or Downstream
Port, which may come handy for hotplugging another device.
{26} Another remark (important to me) in this section: the document
doesn't state firmware expectations. It's clear the firmware is expected
to reserve no IO space for PCI Express Downstream Ports and Root Ports,
but what about MMIO?
We discussed this at length with Alex, but I think we didn't conclude
anything. It would be nice if firmware received some instructions from
this document in this regard, even before we implement our own ports and
bridges in QEMU.
Hmm, I have no idea what to add here, except:
The firmware is expected to reserve at least 2M for each pci bridge?
Just ignore {26} for now please. I'll come back to this later when we
have our own (generic) ports.
+5.3 Hot plug example:
+Using HMP: (add -monitor stdio to QEMU command line)
+ device_add ,id=,bus=
{27} I think the bus=<...> part is incorrect here. Based on the rest of
the guidelines, we have to specify the ID of:
- a PCI Express Root Port, or
- a PCI Express Downstream Port, or
- a PCI-PCI bridge.
I don't get it, you specify what you wrote above as the bus, right?
For example if you start the machine with
-device ioh3420,id=root_port1,
you hotplug with: device_add e1000,bus=root_port1.
My point is that your example names
bus=
which suggests that the following bus *types* can receive hotplugged
devices:
(a) main root bus ("pcie.0")
(c) root port ("PCI Express Root Port Id")
(c) PCI-PCI bridge ("PCI-PCI bridge Id")
(d) extra root bus ("pxb-pcie Id")
Based on the rest of the guidelines, suggestions (a) and (d) are invalid
-- those bus types cannot accept hotplugged devices --, plus option (e)
is missing, namely:
(e) downstream port.
In other words, your example IDs *infer* bus types, and the set of bus
types inferred is both wrong (incorrect elements) and incomplete (one
correct element missing).
Therefore my proposal is to provide the following example:
bus=
And now I understood the 'bug' :)
Thanks,
Marcel
That's all.
Cheers
Laszlo
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/27/16 13:27, Marcel Apfelbaum wrote:
> On 10/14/2016 02:36 PM, Laszlo Ersek wrote:
>> On 10/13/16 16:05, Marcel Apfelbaum wrote:
>>> On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote:
+ -device
pci-bridge,id=pci_bridge1,bus=dmi_pci_bridge1[,chassis_nr=x][,addr=y]
\
+ -device ,bus=pci_bridge1[,addr=x]
>>
>> {13} It would be nice to spell out the valid device addresses (y and x)
>> here too -- can we use 0 for them? SHPC again?
>>
>> Can we / should we simply go with >=1 device addresses?
>>
>
> For pci-bridges only - yes. A better idea (I think) is to disable SHPC
> by default
> from the next QEMU version. Make this slot usable. Sounds OK?
Sure.
+Don't use PCI Express Switches if you don't have too, they use
+an extra PCI bus that may handy to plug another device id it
comes to it.
+
>>
>> {25} I'd put it as: Don't use PCI Express Switches if you don't have
>> too, each one of those uses an extra PCI bus (for its Upstream Port)
>> that could be put to better use with another Root Port or Downstream
>> Port, which may come handy for hotplugging another device.
>>
>> {26} Another remark (important to me) in this section: the document
>> doesn't state firmware expectations. It's clear the firmware is expected
>> to reserve no IO space for PCI Express Downstream Ports and Root Ports,
>> but what about MMIO?
>>
>> We discussed this at length with Alex, but I think we didn't conclude
>> anything. It would be nice if firmware received some instructions from
>> this document in this regard, even before we implement our own ports and
>> bridges in QEMU.
>>
>
> Hmm, I have no idea what to add here, except:
> The firmware is expected to reserve at least 2M for each pci bridge?
Just ignore {26} for now please. I'll come back to this later when we
have our own (generic) ports.
+5.3 Hot plug example:
+Using HMP: (add -monitor stdio to QEMU command line)
+ device_add ,id=,bus=>>> Id/PCI-PCI bridge Id/pxb-pcie Id>
>>
>> {27} I think the bus=<...> part is incorrect here. Based on the rest of
>> the guidelines, we have to specify the ID of:
>> - a PCI Express Root Port, or
>> - a PCI Express Downstream Port, or
>> - a PCI-PCI bridge.
>>
>
> I don't get it, you specify what you wrote above as the bus, right?
> For example if you start the machine with
> -device ioh3420,id=root_port1,
> you hotplug with: device_add e1000,bus=root_port1.
My point is that your example names
bus=
which suggests that the following bus *types* can receive hotplugged
devices:
(a) main root bus ("pcie.0")
(c) root port ("PCI Express Root Port Id")
(c) PCI-PCI bridge ("PCI-PCI bridge Id")
(d) extra root bus ("pxb-pcie Id")
Based on the rest of the guidelines, suggestions (a) and (d) are invalid
-- those bus types cannot accept hotplugged devices --, plus option (e)
is missing, namely:
(e) downstream port.
In other words, your example IDs *infer* bus types, and the set of bus
types inferred is both wrong (incorrect elements) and incomplete (one
correct element missing).
Therefore my proposal is to provide the following example:
bus=
That's all.
Cheers
Laszlo
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/17/2016 05:18 PM, Andrea Bolognani wrote: On Thu, 2016-10-13 at 17:05 +0300, Marcel Apfelbaum wrote: +PCI EXPRESS GUIDELINES +== + +1. Introduction + +The doc proposes best practices on how to use PCI Express/PCI device +in PCI Express based machines and explains the reasoning behind them. + + +2. Device placement strategy + +QEMU does not have a clear socket-device matching mechanism +and allows any PCI/PCI Express device to be plugged into any PCI/PCI Express slot. +Plugging a PCI device into a PCI Express slot might not always work and +is weird anyway since it cannot be done for "bare metal". +Plugging a PCI Express device into a PCI slot will hide the Extended +Configuration Space thus is also not recommended. + +The recommendation is to separate the PCI Express and PCI hierarchies. +PCI Express devices should be plugged only into PCI Express Root Ports and +PCI Express Downstream ports. + +2.1 Root Bus (pcie.0) += +Place only the following kinds of devices directly on the Root Complex: +(1) Devices with dedicated, specific functionality (network card, +graphics card, IDE controller, etc); place only legacy PCI devices on +the Root Complex. These will be considered Integrated Endpoints. +Note: Integrated devices are not hot-pluggable. Hi Andrea, Thanks for the review. s/Integrated devices/Integrated Endpoints/ (which I assume is a Spec-Originated Term) in the last sentence, to be consistent with the one right before it. OK I'm also not sure what you mean by devices with "dedicated, specific functionality", and unfortunately the examples don't seem to be helping me. I'll try to re-phrase a little, the PCI Express spec doesn't say anything specific about what Endpoints can be (as far as I know) and what is happening is that on-board devices remain *usually* PCI and not PCI express.They may be NICs, Sound cards.. Sounds better? +Although the PCI Express spec does not forbid PCI Express devices as +Integrated Endpoints, existing hardware mostly integrates legacy PCI +devices with the Root Complex. Guest OSes are suspected to behave +strangely when PCI Express devices are integrated with the Root Complex. + +(2) PCI Express Root Ports (ioh3420), for starting exclusively PCI Express +hierarchies. + +(3) DMI-PCI bridges (i82801b11-bridge), for starting legacy PCI hierarchies. + +(4) Extra Root Complexes (pxb-pcie), if multiple PCIe Root Buses are needed. + + pcie.0 bus + - +||| | + --- -- -- -- + | PCI Dev | | PCIe Root Port | | DMI-PCI bridge | | pxb-pcie | + --- -- -- -- + +2.1.1 To plug a device into a pcie.0 as Root Complex Integrated Device use: s/Root Complex Integrated Device/Integrated Endpoint/ ? sure I don't know how much any of these terms can be used interchangeably, but it would be good IMHO if we could choose a single term and stick to it throughout the document. + -device [,bus=pcie.0] Is the bus=pcie.0 bit really optional? Will QEMU just assign devices to pcie.0 automatically unless you provide an explicit bus= option telling it otherwise? yes, that is if you dont have a pxb-pcie devices that exposes a new PCI Root Bus. If you have only one PCI Root Bus is OK. +2.1.2 To expose a new PCI Express Root Bus use: + -device pxb-pcie,id=pcie.1,bus_nr=x,[numa_node=y],[addr=z] + Only PCI Express Root Ports and DMI-PCI bridges can be connected to the pcie.1 bus: + -device ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \ + -device i82801b11-bridge,id=dmi_pci_bridge1,bus=pcie.1 Here I really can't see how the bus= option would be optional for ioh3420 but mandatory for i82801b11-bridge. As explained above, is not related to the DMI-PCI bridge, but to the fact we have 2 PCI root buses, so we have to specify which one we need. Neither for above nor for i82801b11-bridge you show that the slot= option exists. Because we don't have this option. Of course these are merely usage examples and are not intended to replace the full documentation: since this is the case, I think we should make that very explicit and possibly avoid listing options we don't use at all, eg. I will mention that the document does not list all the options, only the ones needed for the examples. -device e1000,bus=pcie.0 This will plug a e1000 Ethernet adapter into pcie.0 as an Integrated Endpoint. and so on. Right, but the bus is optional if you don't have multiple PCI root buses (pxb/pxb-pcie). +2.2 PCI Express only hierarchy +== +Always use PC
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/14/2016 02:36 PM, Laszlo Ersek wrote:
On 10/13/16 16:05, Marcel Apfelbaum wrote:
On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote:
Proposes best practices on how to use PCI Express/PCI device
in PCI Express based machines and explain the reasoning behind them.
Signed-off-by: Marcel Apfelbaum
---
Hi,
I am sending the doc twice, it appears the first time didn't make it
to qemu-devel list.
Hi,
Adding people to CC. Sorry for the earlier noise.
Thanks,
Marcel
Hi Laszlo,
Thanks for the review, I'll do my best to address all
the comments in nest version.
[...]
+
+2.2.1 Plugging a PCI Express device into a PCI Express Root Port:
+ -device
ioh3420,id=root_port1,chassis=x[,bus=pcie.0][,slot=y][,addr=z] \
+ -device ,bus=root_port1
+ Note that chassis parameter is compulsory, and must be unique
+ for each PCI Express Root Port.
{7} Hmmm, I think it's rather that the (chassis, slot) *pair* must be
unique. You can get away with leaving the default chassis=0 unspecified,
and spell out just the slots, I think.
Yes you can, I wanted to let the "slot" out of the equation and use the chassis
as a known parameter from the PCI bridge - easier to digest.
However your comment make sense, I'll update the doc.
+2.2.2 Using multi-function PCI Express Root Ports:
+ -device
ioh3420,id=root_port1,multifunction=on,chassis=x[,bus=pcie.0][,slot=y][,addr=z.0]
\
+ -device
ioh3420,id=root_port2,,chassis=x1[,bus=pcie.0][,slot=y1][,addr=z.1] \
+ -device
ioh3420,id=root_port3,,chassis=x2[,bus=pcie.0][,slot=y2][,addr=z.2] \
{8} This looks good to me, except for the double-comma typos: ",,".
+2.2.2 Plugging a PCI Express device into a Switch:
+ -device
ioh3420,id=root_port1,chassis=x[,bus=pcie.0][,slot=y][,addr=z] \
+ -device
x3130-upstream,id=upstream_port1,bus=root_port1[,addr=x] \
+ -device
xio3130-downstream,id=downstream_port1,bus=upstream_port1,chassis=x1[,slot=y1][,addr=z1]]
\
+ -device ,bus=downstream_port1
+
{9} For all of these command lines, can you specify if z=0 (that is,
device#0) is valid or not in the addr=z properties?
Address 0 is valid for all PCI bridges (we see PCI Express
Root/Upstream/Downstream/Ports as PCI bridges)
that do not have an SHPC component.
Even the "regular" pci bridge can use slot 0 if we pass shpc=off as parameter.
For PCI Express slot 0 is always valid, this is why I didn't add it to the doc.
Earlier discussion on this question:
On 10/04/16 18:25, Laine Stump wrote:
On 10/04/2016 11:45 AM, Alex Williamson wrote:
Same with the restriction from using slot
0 on PCI bridges, there's no basis for that except on the root bus.
I tried allowing devices to be plugged into slot 0 of a pci-bridge in
libvirt - qemu barfed, so I moved the "minSlot" for pci-bridge back up
to 1. Slot 0 is completely usable on a dmi-to-pci-bridge though (and
libvirt allows it). At this point, even if qemu enabled using slot 0
of a pci-bridge, libvirt wouldn't be able to expose that to users
(unless the min/max slot of each PCI controller was made visible
somewhere via QMP)
On 10/05/16 12:03, Marcel Apfelbaum wrote:
The reason for not being able to plug a device into slot 0 of a PCI
Bridge is the SHPC (Hot-plug controller) device embedded in the PCI
bridge by default. The SHPC spec requires this. If one disables it
with shpc=false, he should be able to use the slot 0.
For simplicity's sake I guess we should just recommend >=1 slot numbers.
Is a pity to skip slot 0 for PCI Express bridges.
- Root ports and Downstream ports have only slot 0 :)
- Root Complexes (pice.0 and pxb-pcies) can use slot 0
Back to the patch:
On 10/13/16 16:05, Marcel Apfelbaum wrote:
On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote:
+
+2.3 PCI only hierarchy
+==
+Legacy PCI devices can be plugged into pcie.0 as Integrated Devices.
+Besides that use DMI-PCI bridges (i82801b11-bridge) to start PCI
hierarchies.
+
+Prefer flat hierarchies. For most scenarios a single DMI-PCI bridge
(having 32 slots)
+and several PCI-PCI bridges attached to it (each supporting also 32
slots) will support
+hundreds of legacy devices. The recommendation is to populate one
PCI-PCI bridge
+under the DMI-PCI bridge until is full and then plug a new PCI-PCI
bridge...
+
+ pcie.0 bus
+ --
+||
+ --- --
+ | PCI Dev | | DMI-PCI BRIDGE |
+ ----
+ ||
+-----
+| PCI Dev || PCI-PCI Bridge |
+-----
+ | |
+ --- ---
+ | PCI Dev | | PCI Dev |
+
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/17/2016 05:07 PM, Laszlo Ersek wrote:
On 10/17/16 14:07, Gerd Hoffmann wrote:
Hi,
{26} Another remark (important to me) in this section: the document
doesn't state firmware expectations. It's clear the firmware is expected
to reserve no IO space for PCI Express Downstream Ports and Root Ports,
but what about MMIO?
We discussed this at length with Alex, but I think we didn't conclude
anything. It would be nice if firmware received some instructions from
this document in this regard, even before we implement our own ports and
bridges in QEMU.
Where do we stand in terms of generic pcie ports btw?
"planning phase", AFAIR
Indeed, I'll start working on it once this doc is finished.
Thanks,
Marcel
I think the plan is still to communicate suggestions to the firmware via
pci config space, either by using reset defaults of the limit register,
or of that doesn't work due to initialization order issues using some
vendor specific pcie capability.
As long as we don't have that there is nothing do document, other than
maybe briefly mentioning the plans we have and documenting the current
state (2M mmio in seabios, and I think the same for ovmf).
The patches adding the generic ports can also update the documentation
of course.
If we think such recommendations are out of scope at this point, *and*
noone disagrees strongly (Gerd?), then I could add some experimental
fw_cfg knobs to OVMF for this, such as (units in MB):
Why? Given that the virtio mmio bar size issue is solved I don't see a
strong reason to hurry with this. Just wait until the generic ports are
there.
Fine.
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On Mon, 2016-10-17 at 16:26 +0200, Laszlo Ersek wrote: > > > > +2.1 Root Bus (pcie.0) > > > > += > > > > +Place only the following kinds of devices directly on the Root Complex: > > > > +(1) Devices with dedicated, specific functionality (network card, > > > > +graphics card, IDE controller, etc); place only legacy PCI > > > > devices on > > > > +the Root Complex. These will be considered Integrated > > > > Endpoints. > > > > +Note: Integrated devices are not hot-pluggable. > > > > s/Integrated devices/Integrated Endpoints/ (which I assume > > is a Spec-Originated Term) in the last sentence, to be > > consistent with the one right before it. > > > > I'm also not sure what you mean by devices with "dedicated, > > specific functionality", and unfortunately the examples don't > > seem to be helping me. > > That language is from me (I suggested it earlier). It means devices that > you want to use for some actual task, rather than just to build your PCI > (Express) hierarchy with them. "Leaf nodes" or whatever. "Everything > non-bridge". Isn't that basically the definition of Endpoint? It certainly is what I convinced myself "Endpoint" means :) Either way, I think we should spend a few more words there, for clarity's sake. Rewording the phrase to put more emphasis on the "legacy PCI devices only" part would probably be warranted as well. -- Andrea Bolognani / Red Hat / Virtualization
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On Thu, 2016-10-13 at 17:05 +0300, Marcel Apfelbaum wrote: > > +PCI EXPRESS GUIDELINES > > +== > > + > > +1. Introduction > > + > > +The doc proposes best practices on how to use PCI Express/PCI device > > +in PCI Express based machines and explains the reasoning behind them. > > + > > + > > +2. Device placement strategy > > + > > +QEMU does not have a clear socket-device matching mechanism > > +and allows any PCI/PCI Express device to be plugged into any PCI/PCI > > Express slot. > > +Plugging a PCI device into a PCI Express slot might not always work and > > +is weird anyway since it cannot be done for "bare metal". > > +Plugging a PCI Express device into a PCI slot will hide the Extended > > +Configuration Space thus is also not recommended. > > + > > +The recommendation is to separate the PCI Express and PCI hierarchies. > > +PCI Express devices should be plugged only into PCI Express Root Ports and > > +PCI Express Downstream ports. > > + > > +2.1 Root Bus (pcie.0) > > += > > +Place only the following kinds of devices directly on the Root Complex: > > +(1) Devices with dedicated, specific functionality (network card, > > +graphics card, IDE controller, etc); place only legacy PCI devices > > on > > +the Root Complex. These will be considered Integrated Endpoints. > > +Note: Integrated devices are not hot-pluggable. s/Integrated devices/Integrated Endpoints/ (which I assume is a Spec-Originated Term) in the last sentence, to be consistent with the one right before it. I'm also not sure what you mean by devices with "dedicated, specific functionality", and unfortunately the examples don't seem to be helping me. > > +Although the PCI Express spec does not forbid PCI Express devices > > as > > +Integrated Endpoints, existing hardware mostly integrates legacy > > PCI > > +devices with the Root Complex. Guest OSes are suspected to behave > > +strangely when PCI Express devices are integrated with the Root > > Complex. > > + > > +(2) PCI Express Root Ports (ioh3420), for starting exclusively PCI > > Express > > +hierarchies. > > + > > +(3) DMI-PCI bridges (i82801b11-bridge), for starting legacy PCI > > hierarchies. > > + > > +(4) Extra Root Complexes (pxb-pcie), if multiple PCIe Root Buses are > > needed. > > + > > + pcie.0 bus > > + > > - > > +||| | > > + --- -- -- -- > > + | PCI Dev | | PCIe Root Port | | DMI-PCI bridge | | pxb-pcie | > > + --- -- -- -- > > + > > +2.1.1 To plug a device into a pcie.0 as Root Complex Integrated Device use: s/Root Complex Integrated Device/Integrated Endpoint/ ? I don't know how much any of these terms can be used interchangeably, but it would be good IMHO if we could choose a single term and stick to it throughout the document. > > + -device [,bus=pcie.0] Is the bus=pcie.0 bit really optional? Will QEMU just assign devices to pcie.0 automatically unless you provide an explicit bus= option telling it otherwise? > > +2.1.2 To expose a new PCI Express Root Bus use: > > + -device pxb-pcie,id=pcie.1,bus_nr=x,[numa_node=y],[addr=z] > > + Only PCI Express Root Ports and DMI-PCI bridges can be connected to > > the pcie.1 bus: > > + -device > > ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \ > > + -device i82801b11-bridge,id=dmi_pci_bridge1,bus=pcie.1 Here I really can't see how the bus= option would be optional for ioh3420 but mandatory for i82801b11-bridge. Neither for above nor for i82801b11-bridge you show that the slot= option exists. Of course these are merely usage examples and are not intended to replace the full documentation: since this is the case, I think we should make that very explicit and possibly avoid listing options we don't use at all, eg. -device e1000,bus=pcie.0 This will plug a e1000 Ethernet adapter into pcie.0 as an Integrated Endpoint. and so on. > > +2.2 PCI Express only hierarchy > > +== > > +Always use PCI Express Root Ports to start PCI Express hierarchies. > > + > > +A PCI Express Root bus supports up to 32 devices. Since each > > +PCI Express Root Port is a function and a multi-function > > +device may support up to 8 functions, the maximum possible > > +PCI Express Root Ports per PCI Express Root Bus is 256. > > + > > +Prefer coupling PCI Express Root Ports into multi-function devices s/coupling/grouping/ > > +to keep a simple flat hierarchy that is enough for most scenarios. > > +Only use PCI Express Switches (x3130-upstream, xio3130-downstream) > > +if there is no more room for PCI Express Root Ports.
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/17/16 14:07, Gerd Hoffmann wrote:
> Hi,
>
>> {26} Another remark (important to me) in this section: the document
>> doesn't state firmware expectations. It's clear the firmware is expected
>> to reserve no IO space for PCI Express Downstream Ports and Root Ports,
>> but what about MMIO?
>>
>> We discussed this at length with Alex, but I think we didn't conclude
>> anything. It would be nice if firmware received some instructions from
>> this document in this regard, even before we implement our own ports and
>> bridges in QEMU.
>
> Where do we stand in terms of generic pcie ports btw?
"planning phase", AFAIR
>
> I think the plan is still to communicate suggestions to the firmware via
> pci config space, either by using reset defaults of the limit register,
> or of that doesn't work due to initialization order issues using some
> vendor specific pcie capability.
>
> As long as we don't have that there is nothing do document, other than
> maybe briefly mentioning the plans we have and documenting the current
> state (2M mmio in seabios, and I think the same for ovmf).
>
> The patches adding the generic ports can also update the documentation
> of course.
>
>>
>>
>> If we think such recommendations are out of scope at this point, *and*
>> noone disagrees strongly (Gerd?), then I could add some experimental
>> fw_cfg knobs to OVMF for this, such as (units in MB):
>
> Why? Given that the virtio mmio bar size issue is solved I don't see a
> strong reason to hurry with this. Just wait until the generic ports are
> there.
Fine.
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/17/16 16:18, Andrea Bolognani wrote: > On Thu, 2016-10-13 at 17:05 +0300, Marcel Apfelbaum wrote: >>> +PCI EXPRESS GUIDELINES >>> +== >>> + >>> +1. Introduction >>> + >>> +The doc proposes best practices on how to use PCI Express/PCI device >>> +in PCI Express based machines and explains the reasoning behind them. >>> + >>> + >>> +2. Device placement strategy >>> + >>> +QEMU does not have a clear socket-device matching mechanism >>> +and allows any PCI/PCI Express device to be plugged into any PCI/PCI >>> Express slot. >>> +Plugging a PCI device into a PCI Express slot might not always work and >>> +is weird anyway since it cannot be done for "bare metal". >>> +Plugging a PCI Express device into a PCI slot will hide the Extended >>> +Configuration Space thus is also not recommended. >>> + >>> +The recommendation is to separate the PCI Express and PCI hierarchies. >>> +PCI Express devices should be plugged only into PCI Express Root Ports and >>> +PCI Express Downstream ports. >>> + >>> +2.1 Root Bus (pcie.0) >>> += >>> +Place only the following kinds of devices directly on the Root Complex: >>> +(1) Devices with dedicated, specific functionality (network card, >>> +graphics card, IDE controller, etc); place only legacy PCI devices >>> on >>> +the Root Complex. These will be considered Integrated Endpoints. >>> +Note: Integrated devices are not hot-pluggable. > > s/Integrated devices/Integrated Endpoints/ (which I assume > is a Spec-Originated Term) in the last sentence, to be > consistent with the one right before it. > > I'm also not sure what you mean by devices with "dedicated, > specific functionality", and unfortunately the examples don't > seem to be helping me. That language is from me (I suggested it earlier). It means devices that you want to use for some actual task, rather than just to build your PCI (Express) hierarchy with them. "Leaf nodes" or whatever. "Everything non-bridge". Thanks Laszlo
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
Hi,
> {26} Another remark (important to me) in this section: the document
> doesn't state firmware expectations. It's clear the firmware is expected
> to reserve no IO space for PCI Express Downstream Ports and Root Ports,
> but what about MMIO?
>
> We discussed this at length with Alex, but I think we didn't conclude
> anything. It would be nice if firmware received some instructions from
> this document in this regard, even before we implement our own ports and
> bridges in QEMU.
Where do we stand in terms of generic pcie ports btw?
I think the plan is still to communicate suggestions to the firmware via
pci config space, either by using reset defaults of the limit register,
or of that doesn't work due to initialization order issues using some
vendor specific pcie capability.
As long as we don't have that there is nothing do document, other than
maybe briefly mentioning the plans we have and documenting the current
state (2M mmio in seabios, and I think the same for ovmf).
The patches adding the generic ports can also update the documentation
of course.
>
>
> If we think such recommendations are out of scope at this point, *and*
> noone disagrees strongly (Gerd?), then I could add some experimental
> fw_cfg knobs to OVMF for this, such as (units in MB):
Why? Given that the virtio mmio bar size issue is solved I don't see a
strong reason to hurry with this. Just wait until the generic ports are
there.
cheers,
Gerd
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/13/16 16:05, Marcel Apfelbaum wrote:
> On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote:
>> Proposes best practices on how to use PCI Express/PCI device
>> in PCI Express based machines and explain the reasoning behind them.
>>
>> Signed-off-by: Marcel Apfelbaum
>> ---
>>
>> Hi,
>>
>> I am sending the doc twice, it appears the first time didn't make it
>> to qemu-devel list.
>
> Hi,
>
> Adding people to CC. Sorry for the earlier noise.
>
> Thanks,
> Marcel
>
>>
>> RFC->v2:
>> - Addressed a lot of comments from the reviewers (many thanks to all,
>> especially to Laszlo)
>>
>> Since the RFC mail-thread was relatively long and already
>> has passed a lot of time from the RFC, I post this version
>> even if is very possible that I left some of the comments out,
>> my apologies if so.
>>
>> I will go over the comments again, in the meantime please
>> feel free to comment on this version, even if on something
>> you've already pointed out.
>>
>> It may take a day or two until I'll be able to respond, but I
>> will do my best to address all comments.
>>
>> Thanks,
>> Marcel
>>
>>
>> docs/pcie.txt | 273
>> ++
>> 1 file changed, 273 insertions(+)
>> create mode 100644 docs/pcie.txt
>>
>> diff --git a/docs/pcie.txt b/docs/pcie.txt
>> new file mode 100644
>> index 000..7d852f1
>> --- /dev/null
>> +++ b/docs/pcie.txt
>> @@ -0,0 +1,273 @@
>> +PCI EXPRESS GUIDELINES
>> +==
>> +
>> +1. Introduction
>> +
>> +The doc proposes best practices on how to use PCI Express/PCI device
>> +in PCI Express based machines and explains the reasoning behind them.
>> +
>> +
>> +2. Device placement strategy
>> +
>> +QEMU does not have a clear socket-device matching mechanism
>> +and allows any PCI/PCI Express device to be plugged into any PCI/PCI
>> Express slot.
{1} This line seems too long; I suggest to rewrap the entire document at
79 chars (except command line fragments where the wrapping would be
impractical).
>> +Plugging a PCI device into a PCI Express slot might not always work and
>> +is weird anyway since it cannot be done for "bare metal".
>> +Plugging a PCI Express device into a PCI slot will hide the Extended
>> +Configuration Space thus is also not recommended.
>> +
>> +The recommendation is to separate the PCI Express and PCI hierarchies.
>> +PCI Express devices should be plugged only into PCI Express Root
>> Ports and
>> +PCI Express Downstream ports.
>> +
>> +2.1 Root Bus (pcie.0)
>> +=
>> +Place only the following kinds of devices directly on the Root Complex:
>> +(1) Devices with dedicated, specific functionality (network card,
>> +graphics card, IDE controller, etc); place only legacy PCI
{2} I recommend to replace the semicolon (;) with a colon (:) here.
>> devices on
>> +the Root Complex. These will be considered Integrated Endpoints.
>> +Note: Integrated devices are not hot-pluggable.
>> +
>> +Although the PCI Express spec does not forbid PCI Express
>> devices as
>> +Integrated Endpoints, existing hardware mostly integrates
>> legacy PCI
>> +devices with the Root Complex. Guest OSes are suspected to
>> behave
>> +strangely when PCI Express devices are integrated with the
>> Root Complex.
>> +
>> +(2) PCI Express Root Ports (ioh3420), for starting exclusively
>> PCI Express
>> +hierarchies.
>> +
>> +(3) DMI-PCI bridges (i82801b11-bridge), for starting legacy PCI
>> hierarchies.
>> +
>> +(4) Extra Root Complexes (pxb-pcie), if multiple PCIe Root Buses
>> are needed.
{3} s/PCIe/PCI Express/, please :)
>> +
>> + pcie.0 bus
>> +
>> -
>>
>> +||| |
>> + --- -- --
>> --
>> + | PCI Dev | | PCIe Root Port | | DMI-PCI bridge | |
>> pxb-pcie |
>> + --- -- --
>> --
>> +
>> +2.1.1 To plug a device into a pcie.0 as Root Complex Integrated
{4} s/a pcie.0 as/pcie.0 as a/
>> Device use:
>> + -device [,bus=pcie.0]
>> +2.1.2 To expose a new PCI Express Root Bus use:
>> + -device pxb-pcie,id=pcie.1,bus_nr=x,[numa_node=y],[addr=z]
{5} I think the commas should be moved into the brackets (like below).
>> + Only PCI Express Root Ports and DMI-PCI bridges can be
>> connected to the pcie.1 bus:
>> + -device
>> ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \
>> + -device i82801b11-bridge,id=dmi_pci_bridge1,bus=pcie.1
>> +
>> +
>> +2.2 PCI Express only hierarchy
>> +==
>> +Always use PCI Express Root Ports to start PCI Express hierarchies.
>> +
>> +A PCI Express Root bus supports up to 32 devices. Since each
>> +PCI Express Root Port is a function
Re: [Qemu-devel] [PATCH V2 RESEND] docs: add PCIe devices placement guidelines
On 10/13/2016 04:52 PM, Marcel Apfelbaum wrote: Proposes best practices on how to use PCI Express/PCI device in PCI Express based machines and explain the reasoning behind them. Signed-off-by: Marcel Apfelbaum --- Hi, I am sending the doc twice, it appears the first time didn't make it to qemu-devel list. Hi, Adding people to CC. Sorry for the earlier noise. Thanks, Marcel RFC->v2: - Addressed a lot of comments from the reviewers (many thanks to all, especially to Laszlo) Since the RFC mail-thread was relatively long and already has passed a lot of time from the RFC, I post this version even if is very possible that I left some of the comments out, my apologies if so. I will go over the comments again, in the meantime please feel free to comment on this version, even if on something you've already pointed out. It may take a day or two until I'll be able to respond, but I will do my best to address all comments. Thanks, Marcel docs/pcie.txt | 273 ++ 1 file changed, 273 insertions(+) create mode 100644 docs/pcie.txt diff --git a/docs/pcie.txt b/docs/pcie.txt new file mode 100644 index 000..7d852f1 --- /dev/null +++ b/docs/pcie.txt @@ -0,0 +1,273 @@ +PCI EXPRESS GUIDELINES +== + +1. Introduction + +The doc proposes best practices on how to use PCI Express/PCI device +in PCI Express based machines and explains the reasoning behind them. + + +2. Device placement strategy + +QEMU does not have a clear socket-device matching mechanism +and allows any PCI/PCI Express device to be plugged into any PCI/PCI Express slot. +Plugging a PCI device into a PCI Express slot might not always work and +is weird anyway since it cannot be done for "bare metal". +Plugging a PCI Express device into a PCI slot will hide the Extended +Configuration Space thus is also not recommended. + +The recommendation is to separate the PCI Express and PCI hierarchies. +PCI Express devices should be plugged only into PCI Express Root Ports and +PCI Express Downstream ports. + +2.1 Root Bus (pcie.0) += +Place only the following kinds of devices directly on the Root Complex: +(1) Devices with dedicated, specific functionality (network card, +graphics card, IDE controller, etc); place only legacy PCI devices on +the Root Complex. These will be considered Integrated Endpoints. +Note: Integrated devices are not hot-pluggable. + +Although the PCI Express spec does not forbid PCI Express devices as +Integrated Endpoints, existing hardware mostly integrates legacy PCI +devices with the Root Complex. Guest OSes are suspected to behave +strangely when PCI Express devices are integrated with the Root Complex. + +(2) PCI Express Root Ports (ioh3420), for starting exclusively PCI Express +hierarchies. + +(3) DMI-PCI bridges (i82801b11-bridge), for starting legacy PCI hierarchies. + +(4) Extra Root Complexes (pxb-pcie), if multiple PCIe Root Buses are needed. + + pcie.0 bus + - +||| | + --- -- -- -- + | PCI Dev | | PCIe Root Port | | DMI-PCI bridge | | pxb-pcie | + --- -- -- -- + +2.1.1 To plug a device into a pcie.0 as Root Complex Integrated Device use: + -device [,bus=pcie.0] +2.1.2 To expose a new PCI Express Root Bus use: + -device pxb-pcie,id=pcie.1,bus_nr=x,[numa_node=y],[addr=z] + Only PCI Express Root Ports and DMI-PCI bridges can be connected to the pcie.1 bus: + -device ioh3420,id=root_port1[,bus=pcie.1][,chassis=x][,slot=y][,addr=z] \ + -device i82801b11-bridge,id=dmi_pci_bridge1,bus=pcie.1 + + +2.2 PCI Express only hierarchy +== +Always use PCI Express Root Ports to start PCI Express hierarchies. + +A PCI Express Root bus supports up to 32 devices. Since each +PCI Express Root Port is a function and a multi-function +device may support up to 8 functions, the maximum possible +PCI Express Root Ports per PCI Express Root Bus is 256. + +Prefer coupling PCI Express Root Ports into multi-function devices +to keep a simple flat hierarchy that is enough for most scenarios. +Only use PCI Express Switches (x3130-upstream, xio3130-downstream) +if there is no more room for PCI Express Root Ports. +Please see section 4. for further justifications. + +Plug only PCI Express devices into PCI Express Ports. + + + pcie.0 bus + -- +| || + --- + | Root Port || Root
