Re: [edk2] PCIe hotplug into downstream port

[Bill Paul]

Of all the gin joints in all the towns in all the world, Laszlo Ersek had to 
walk into mine at 11:58:29 on Thursday 30 June 2016 and say:

> On 06/30/16 20:14, Brian J. Johnson wrote:
> > On 06/30/2016 11:47 AM, Laszlo Ersek wrote:
> >> On 06/30/16 18:39, Marcel Apfelbaum wrote:
> >>> On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote:
>  On 06/30/2016 04:07 PM, Laszlo Ersek wrote:
> > Hi,
[snip] 
> > We can safely skip
> > allocating IO ports to those cards, saving significant space in the
> > overall IO port map.  We've worked with the card vendors on cleaning up
> > their PCIe advertisements and eliminating the use of IO ports, but it
> > takes time.
> 
> I surprisingly often hear about non-conformance or unjustified
> behavior... I guess with huge, elaborate specs, this is unavoidable. No
> single programmer might be able to internalize it all.

This is one of my pet peeves. Things that've thrown me for a loop are:

- Customers with their own special PCI devices that want _enormous_ amounts of 
MMIO space -- there was one case recently where the device wanted an 8GB 
window. (Even if this is legal according to the spec, it strikes me as a 
little unfriendly.)

- Oddball PCI/PCIe controller implementations in SoCs. For example, some 
devices implement support for MSI but don't support the ability to dynamically 
allocate interrupt vectors for multiple MSI sources. This means there's often 
just one vector for MSI events and it's shared by all devices on the bus. (The 
Freescale/NXP i.MX6 is one example of this.)

- FPGA-based PCIe controller logic blocks. To be fair, I've only worked with 
one of these, namely the Xilinx Zynq7k. Maybe that one is just particularly 
irritating. Out of the box, the Zynq doesn't support PCIe: you have to load a 
bitstream file into it to make the internal FPGA think it's a PCIe controller. 
I think it's the case that the customer can customize the logic to add or 
remove features, probably to optimize the use of the available logic gates in 
the FPGA. Unfortunately this can make driver development difficult because it 
makes the "hardware" a moving target. It also means the "hardware" design 
might be kept very simple and a lot of heavy lifting is deferred to the driver 
software.

The Zynq PCIe controller has only one interrupt vector, and it uses it for 
_all_ events (INTx interrupts, MSI interrupts and controller errors). For the 
INTx interrupts, you have to implement level-triggered interrupt semantics in 
software. If you do it wrong, you could miss an event and get totally stuck. 
(And last time I looked at it, I wasn't 100% convinced that the Linux driver 
was doing it exactly right either.) If you only ever want to connect one PCIe 
device to the controller, you can get away with some fairly simple logic, but 
hardly any of our customers would be satisfied with that.

Also, the documentation for the PCIe controller block says that it supports 
both MMIO and I/O space BARs, but the particular bitstream file that I was 
given to work with only seemed to support MMIO. (The registers documented to 
configure the outbound window for I/O transactions didn't seem to do 
anything.)

While I realize that I/O space BARs and INTx interrupts are old and busted and 
MMIO and MSI are the new hotness, we sometimes have customers with highly 
specialized legacy hardware and I would prefer not to throw a spanner in their 
works if I can avoid it.

-Bill

> Thanks!
> Laszlo
> ___
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

-- 
=
-Bill Paul(510) 749-2329 | Senior Member of Technical Staff,
 wp...@windriver.com | Master of Unix-Fu - Wind River Systems
=
   "I put a dollar in a change machine. Nothing changed." - George Carlin
=
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Brian J. Johnson]

On 06/30/2016 11:47 AM, Laszlo Ersek wrote:

On 06/30/16 18:39, Marcel Apfelbaum wrote:

On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote:

On 06/30/2016 04:07 PM, Laszlo Ersek wrote:

Hi,

does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?

The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.

In my understanding:

* Each PCIe downstream port qualifies as a separate bus / bridge.

* Bridges need IO and MMIO resources for themselves. When a device is
behind a bridge at boot, the device's IO and MMIO BARs are carved out
of the bridge's IO and MMIO apertures. For IO, the standard bridge
aperture size is 4096 (byte-sized) ports.

* If no device behind a bridge needs a specific type of resource (for
example, no devices behind a bridge have IO BARs), then that kind of
aperture is not necessary for the bridge either.

A special case of this is when there are no devices behind a bridge
at all. Example: PCIe downstream port empty at boot. In this case,
the bridge needs no resources at all, at boot.

* Different firmwares seem to follow different policies in the last
case (-> bridge empty at boot). SeaBIOS for example allocates the
0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
allocates no apertures for bridges that have no devices behind them
at boot.

This appears to be a trade-off: if the bridge remains empty for the
entire lifetime of the OS, then any IO ports that have been allocated
in advance will be in vain (wasted). However, if the aperture is not
allocated at boot, then hotplug under the bridge may not work --
resources for the device being hot-plugged cannot be allocated from
nonexistent bridge apertures.



Hi Laszlo,
I fully agree with your assessment.

  From the firmware point of view it seems like a waste to allocate an
IO/MEM range for
an unused PCI/PCIe-bridge indeed. Especially IO.

But what about MEM?
Let's have a look to this special case: PCIe downstream ports on
64-bit systems.

1. PCIe spec requires devices to work correctly even without enabling
their IO BARs.
2. 64-bit Memory space is pretty large, decent CPUs support 46-bit
addresses, so even if the system uses Terabytes of RAM
 and has dozens of switches we still have enough unused MEM space.

My point is we can skip assigning IO for PCIe ports but we can assign
a 4K MEM range in 64-bit memory for them
without being too greedy.


I meant 2M MEM range, sorry.  4k is the min range for IO, of course.




One thing to watch out for is prefetchable vs. non-prefetchable MEM 
ranges.  On Intel chipsets, the high (64-bit) MMIO range is always 
prefetchable, which mainly makes it useful for frame buffers, 
accelerator cards with RAM mapped to MMIO in the host, and the like. 
Only the low (32-bit) MMIO space is non-prefetchable, and suitable for 
registers and things which need precise ordering of loads and stores.


So you can demote a prefetchable BAR into non-prefetchable MMIO space if 
necessary, but not the other way around.  Which is a shame, because the 
32-bit MMIO space is much more limited in size than the 64-bit space.



I've just been looking at the values reserved by SeaBIOS [src/fw/pciinit.c]:

#define PCI_BRIDGE_MEM_MIN(1<<21)  // 2M == hugepage size
#define PCI_BRIDGE_IO_MIN  0x1000  // mandated by pci bridge spec

2MB sounds perfectly fine for MEM, especially given that OVMF supports a
64-bit MMIO aperture (32GB in size by default).

We discussed the IO size earlier -- I think 4K is overkill (despite
being required by the standard). We considered half K, 1K, 2K earlier...
I think I'll take a stab at half K first.


It depends on what your hardware (physical or virtual) supports.  A 
standards-compliant bridge may not be able to map I/O ports to targets 
on less than a 4k boundary.  Yes, 4k is ridiculous overkill, but it may 
be your only choice, depending on the details of the h/w and/or VMM.


We've had to add special cases in BIOS for certain cards we support 
which request IO space, but don't actually use it.  We can safely skip 
allocating IO ports to those cards, saving significant space in the 
overall IO port map.  We've worked with the card vendors on cleaning up 
their PCIe advertisements and eliminating the use of IO ports, but it 
takes time.

--

Brian



  "My software never has bugs.  It just develops random features."
   -- Quoted by Khan Tran
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Laszlo Ersek]

On 06/30/16 18:39, Marcel Apfelbaum wrote:
> On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote:
>> On 06/30/2016 04:07 PM, Laszlo Ersek wrote:
>>> Hi,
>>>
>>> does anyone on this list have experience with $SUBJECT, using physical
>>> UEFI firmware derived from edk2?
>>>
>>> The problem we're seeing with OVMF is the following: PCIe hotplug works
>>> with PCIe root ports, but it doesn't work with PCIe downstream ports.
>>> The error messages reported by both Linux and Windows guests indicate
>>> "lack of resources" for the device being plugged.
>>>
>>> In my understanding:
>>>
>>> * Each PCIe downstream port qualifies as a separate bus / bridge.
>>>
>>> * Bridges need IO and MMIO resources for themselves. When a device is
>>>behind a bridge at boot, the device's IO and MMIO BARs are carved out
>>>of the bridge's IO and MMIO apertures. For IO, the standard bridge
>>>aperture size is 4096 (byte-sized) ports.
>>>
>>> * If no device behind a bridge needs a specific type of resource (for
>>>example, no devices behind a bridge have IO BARs), then that kind of
>>>aperture is not necessary for the bridge either.
>>>
>>>A special case of this is when there are no devices behind a bridge
>>>at all. Example: PCIe downstream port empty at boot. In this case,
>>>the bridge needs no resources at all, at boot.
>>>
>>> * Different firmwares seem to follow different policies in the last
>>>case (-> bridge empty at boot). SeaBIOS for example allocates the
>>>0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
>>>allocates no apertures for bridges that have no devices behind them
>>>at boot.
>>>
>>>This appears to be a trade-off: if the bridge remains empty for the
>>>entire lifetime of the OS, then any IO ports that have been allocated
>>>in advance will be in vain (wasted). However, if the aperture is not
>>>allocated at boot, then hotplug under the bridge may not work --
>>>resources for the device being hot-plugged cannot be allocated from
>>>nonexistent bridge apertures.
>>>
>>
>> Hi Laszlo,
>> I fully agree with your assessment.
>>
>>  From the firmware point of view it seems like a waste to allocate an
>> IO/MEM range for
>> an unused PCI/PCIe-bridge indeed. Especially IO.
>>
>> But what about MEM?
>> Let's have a look to this special case: PCIe downstream ports on
>> 64-bit systems.
>>
>> 1. PCIe spec requires devices to work correctly even without enabling
>> their IO BARs.
>> 2. 64-bit Memory space is pretty large, decent CPUs support 46-bit
>> addresses, so even if the system uses Terabytes of RAM
>> and has dozens of switches we still have enough unused MEM space.
>>
>> My point is we can skip assigning IO for PCIe ports but we can assign
>> a 4K MEM range in 64-bit memory for them
>> without being too greedy.
> 
> I meant 2M MEM range, sorry.  4k is the min range for IO, of course.

I've just been looking at the values reserved by SeaBIOS [src/fw/pciinit.c]:

#define PCI_BRIDGE_MEM_MIN(1<<21)  // 2M == hugepage size
#define PCI_BRIDGE_IO_MIN  0x1000  // mandated by pci bridge spec

2MB sounds perfectly fine for MEM, especially given that OVMF supports a
64-bit MMIO aperture (32GB in size by default).

We discussed the IO size earlier -- I think 4K is overkill (despite
being required by the standard). We considered half K, 1K, 2K earlier...
I think I'll take a stab at half K first.

Thanks!
Laszlo
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Marcel Apfelbaum]

On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote:

On 06/30/2016 04:07 PM, Laszlo Ersek wrote:

Hi,

does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?

The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.

In my understanding:

* Each PCIe downstream port qualifies as a separate bus / bridge.

* Bridges need IO and MMIO resources for themselves. When a device is
   behind a bridge at boot, the device's IO and MMIO BARs are carved out
   of the bridge's IO and MMIO apertures. For IO, the standard bridge
   aperture size is 4096 (byte-sized) ports.

* If no device behind a bridge needs a specific type of resource (for
   example, no devices behind a bridge have IO BARs), then that kind of
   aperture is not necessary for the bridge either.

   A special case of this is when there are no devices behind a bridge
   at all. Example: PCIe downstream port empty at boot. In this case,
   the bridge needs no resources at all, at boot.

* Different firmwares seem to follow different policies in the last
   case (-> bridge empty at boot). SeaBIOS for example allocates the
   0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
   allocates no apertures for bridges that have no devices behind them
   at boot.

   This appears to be a trade-off: if the bridge remains empty for the
   entire lifetime of the OS, then any IO ports that have been allocated
   in advance will be in vain (wasted). However, if the aperture is not
   allocated at boot, then hotplug under the bridge may not work --
   resources for the device being hot-plugged cannot be allocated from
   nonexistent bridge apertures.



Hi Laszlo,
I fully agree with your assessment.

 From the firmware point of view it seems like a waste to allocate an IO/MEM 
range for
an unused PCI/PCIe-bridge indeed. Especially IO.

But what about MEM?
Let's have a look to this special case: PCIe downstream ports on 64-bit systems.

1. PCIe spec requires devices to work correctly even without enabling their IO 
BARs.
2. 64-bit Memory space is pretty large, decent CPUs support 46-bit addresses, 
so even if the system uses Terabytes of RAM
and has dozens of switches we still have enough unused MEM space.

My point is we can skip assigning IO for PCIe ports but we can assign a 4K MEM 
range in 64-bit memory for them
without being too greedy.


I meant 2M MEM range, sorry.  4k is the min range for IO, of course.


Thanks,
Marcel



Would this be a decent trade-off *for OVMF* ?

Thanks,
Marcel


And that's what we are experiencing with OVMF.

My questions are:

- Is the above behavior of the edk2 PCI bus driver intentional?

- Does PCIe hotplug into downstream ports work with any (phys) firmware
   forked from edk2? Brian, Samer, do you guys have experience with this?

- What could be the difference between root ports and downstream ports?
   (Hotplug into root ports seems to work fine.) Are OSes entitled to
   allocate any unused address space (MMIO and IO) right when a device is
   hot-plugged into a root port?

Thanks!
Laszlo





___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Marcel Apfelbaum]

On 06/30/2016 04:07 PM, Laszlo Ersek wrote:

Hi,

does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?

The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.

In my understanding:

* Each PCIe downstream port qualifies as a separate bus / bridge.

* Bridges need IO and MMIO resources for themselves. When a device is
   behind a bridge at boot, the device's IO and MMIO BARs are carved out
   of the bridge's IO and MMIO apertures. For IO, the standard bridge
   aperture size is 4096 (byte-sized) ports.

* If no device behind a bridge needs a specific type of resource (for
   example, no devices behind a bridge have IO BARs), then that kind of
   aperture is not necessary for the bridge either.

   A special case of this is when there are no devices behind a bridge
   at all. Example: PCIe downstream port empty at boot. In this case,
   the bridge needs no resources at all, at boot.

* Different firmwares seem to follow different policies in the last
   case (-> bridge empty at boot). SeaBIOS for example allocates the
   0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
   allocates no apertures for bridges that have no devices behind them
   at boot.

   This appears to be a trade-off: if the bridge remains empty for the
   entire lifetime of the OS, then any IO ports that have been allocated
   in advance will be in vain (wasted). However, if the aperture is not
   allocated at boot, then hotplug under the bridge may not work --
   resources for the device being hot-plugged cannot be allocated from
   nonexistent bridge apertures.



Hi Laszlo,
I fully agree with your assessment.

From the firmware point of view it seems like a waste to allocate an IO/MEM 
range for
an unused PCI/PCIe-bridge indeed. Especially IO.

But what about MEM?
Let's have a look to this special case: PCIe downstream ports on 64-bit systems.

1. PCIe spec requires devices to work correctly even without enabling their IO 
BARs.
2. 64-bit Memory space is pretty large, decent CPUs support 46-bit addresses, 
so even if the system uses Terabytes of RAM
   and has dozens of switches we still have enough unused MEM space.

My point is we can skip assigning IO for PCIe ports but we can assign a 4K MEM 
range in 64-bit memory for them
without being too greedy.

Would this be a decent trade-off *for OVMF* ?

Thanks,
Marcel


And that's what we are experiencing with OVMF.

My questions are:

- Is the above behavior of the edk2 PCI bus driver intentional?

- Does PCIe hotplug into downstream ports work with any (phys) firmware
   forked from edk2? Brian, Samer, do you guys have experience with this?

- What could be the difference between root ports and downstream ports?
   (Hotplug into root ports seems to work fine.) Are OSes entitled to
   allocate any unused address space (MMIO and IO) right when a device is
   hot-plugged into a root port?

Thanks!
Laszlo



___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Marcel Apfelbaum]

On 06/30/2016 06:06 PM, Alex Williamson wrote:

On Thu, 30 Jun 2016 15:07:27 +0200
Laszlo Ersek  wrote:

- What could be the difference between root ports and downstream ports?
   (Hotplug into root ports seems to work fine.)


Hi Laszlo,

It should be no difference IMHO. Both are treated the same by the PCIe spec as 
far as I know.

Are OSes entitled to

   allocate any unused address space (MMIO and IO) right when a device is
   hot-plugged into a root port?




Yes, resources "re-banlancing"| can be triggered by the OS and at least Windows 
is actually
doing so if you hot-plug devices behind the bridge using more IO/MEM than what 
is allocated
for the bridge by the firmware. I don't know for sure, but I think Linux will 
try this too.


A possible difference is simply the depth of the hierarchy, the
apertures on a root port come directly from the host bridge and there's
no affect to other devices to disable and resize the root port
apertures.

 In order to resize a switch downstream port aperture, the

OS would need to touch multiple levels, which could affect peer devices
already in operation.


But this should be the same as with nested PCI bridges, right? And If I would 
have to guess,
there is no problem with them.


  Does hotplug to a downstream switch port work if

the hot added device is the only endpoint within that sub-hierarchy?


I don't see why not. If the firmware pre-allocates IO/MEM ranges there is no 
problem.
Otherwise the OS uses the free IO/MEM ranges specified by the _CRS for the 
respective host-bridge.

Thanks,
Marcel

 I

wouldn't necessarily be surprised either way, it seems like a
complicated resource runtime reallocation issue that probably isn't
very prevalent on real hardware.  Thanks,

Alex



___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Alex Williamson]

On Thu, 30 Jun 2016 15:07:27 +0200
Laszlo Ersek  wrote:
> - What could be the difference between root ports and downstream ports?
>   (Hotplug into root ports seems to work fine.) Are OSes entitled to
>   allocate any unused address space (MMIO and IO) right when a device is
>   hot-plugged into a root port?

A possible difference is simply the depth of the hierarchy, the
apertures on a root port come directly from the host bridge and there's
no affect to other devices to disable and resize the root port
apertures.  In order to resize a switch downstream port aperture, the
OS would need to touch multiple levels, which could affect peer devices
already in operation.  Does hotplug to a downstream switch port work if
the hot added device is the only endpoint within that sub-hierarchy?  I
wouldn't necessarily be surprised either way, it seems like a
complicated resource runtime reallocation issue that probably isn't
very prevalent on real hardware.  Thanks,

Alex
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Laszlo Ersek]

On 06/30/16 16:48, Brian J. Johnson wrote:
> On 06/30/2016 09:35 AM, Laszlo Ersek wrote:
>> On 06/30/16 16:24, Brian J. Johnson wrote:
>>> On 06/30/2016 08:07 AM, Laszlo Ersek wrote:
>>
 - Does PCIe hotplug into downstream ports work with any (phys) firmware
 forked from edk2? Brian, Samer, do you guys have experience with
 this?
>>>
>>> Yes, we support it on our UV server line.  Physically, hotplug is only
>>> possible in expansion chassis designed for that purpose.  (That's not a
>>> problem with virtual machines, of course.)  But we've had to do quite a
>>> bit of work on the host bridge driver to allowing scaling to the sizes
>>> of systems we support (hundreds of sockets and PCIe slots, dozens of TB
>>> of RAM), including reserving resources for hotplug only in the specific
>>> cases where it's possible.
>>
>> Thanks a lot for your answer! Could you perhaps check for me if your
>> platform firmware implements EFI_PCI_HOT_PLUG_INIT_PROTOCOL, highlighted
>> by Andrew?
> 
> 
> Yes, we do.  We have a customized implementation of this protocol which
> provides padding values optimized for the expansion chassis we support.

Awesome. That's what I'm looking at in the PI spec right now.

I'm being led to believe that I won't need to return any
EFI_HPC_LOCATIONs from GetRootHpcList(), and correspondingly I can stub
out InitializeRootHpc() as well. The PI spec allows this:

* As stated above, root HPCs may require firmware initialization.
  PI Architecture firmware must be capable of supporting root HPCs
  that are initialized by hardware and do not require any firmware
  initialization.

[...]

Scan all the devices in the specified bus range and the specified
segment, one bus at a time. If the device is a PCI-to-PCI bridge,
update the bus numbers and program the bus number registers in the
PCI-to-PCI bridge hardware. If the device path of a device matches
that of a root HPC and it is not a PCI-to-CardBus bridge, it must
be initialized by calling
EFI_PCI_HOT_PLUG_INIT_PROTOCOL.InitializeRootHpc() before the bus it
controls can be fully enumerated.

[...]

Root HPC

These HPCs must be initialized by calling InitializeRootHpc()
during the enumeration process. These HPCs will also require
resource padding. The platform code must have a priori
knowledge of these devices and must know how to initialize
them. There may not be any way to access their PCI
configuration space before the PCI enumerator programs all the
upstream bridges and thus enables the path to these devices.

The recursive enumeration works just fine already. On QEMU's virtual
hardware, there doesn't seem to be any need for this kind of
initialization. It's the resource padding bit that I have to figure out.

Thank you!
Laszlo
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Brian J. Johnson]

On 06/30/2016 09:35 AM, Laszlo Ersek wrote:

On 06/30/16 16:24, Brian J. Johnson wrote:

On 06/30/2016 08:07 AM, Laszlo Ersek wrote:



- Does PCIe hotplug into downstream ports work with any (phys) firmware
forked from edk2? Brian, Samer, do you guys have experience with this?


Yes, we support it on our UV server line.  Physically, hotplug is only
possible in expansion chassis designed for that purpose.  (That's not a
problem with virtual machines, of course.)  But we've had to do quite a
bit of work on the host bridge driver to allowing scaling to the sizes
of systems we support (hundreds of sockets and PCIe slots, dozens of TB
of RAM), including reserving resources for hotplug only in the specific
cases where it's possible.


Thanks a lot for your answer! Could you perhaps check for me if your
platform firmware implements EFI_PCI_HOT_PLUG_INIT_PROTOCOL, highlighted
by Andrew?



Yes, we do.  We have a customized implementation of this protocol which 
provides padding values optimized for the expansion chassis we support.

--

Brian J. Johnson



  My statements are my own, are not authorized by SGI, and do not
  necessarily represent SGI’s positions.
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Laszlo Ersek]

On 06/30/16 16:24, Brian J. Johnson wrote:
> On 06/30/2016 08:07 AM, Laszlo Ersek wrote:

>> - Does PCIe hotplug into downstream ports work with any (phys) firmware
>>forked from edk2? Brian, Samer, do you guys have experience with this?
> 
> Yes, we support it on our UV server line.  Physically, hotplug is only
> possible in expansion chassis designed for that purpose.  (That's not a
> problem with virtual machines, of course.)  But we've had to do quite a
> bit of work on the host bridge driver to allowing scaling to the sizes
> of systems we support (hundreds of sockets and PCIe slots, dozens of TB
> of RAM), including reserving resources for hotplug only in the specific
> cases where it's possible.

Thanks a lot for your answer! Could you perhaps check for me if your
platform firmware implements EFI_PCI_HOT_PLUG_INIT_PROTOCOL, highlighted
by Andrew?

Thanks!
Laszlo

___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Laszlo Ersek]

On 06/30/16 16:22, Andrew Fish wrote:
> 
>> On Jun 30, 2016, at 6:07 AM, Laszlo Ersek  wrote:
>>
>> Hi,
>>
>> does anyone on this list have experience with $SUBJECT, using physical
>> UEFI firmware derived from edk2?
>>
> 
> Laszlo,
> 
> I don't know the edk2 specific answer from memory, but in general as
> a system gets larger it is harder and harder to efficiently over
> allocate resources. So there is usually some mechanism to configure
> the over allocation of resources and identify which ports support hot
> plug.
> 
> Maybe: 
> https://github.com/tianocore/edk2/blob/master/MdePkg/Include/Protocol/PciHotPlugInit.h

That seems to be exactly what OVMF should implement -- thank you!

Now I just need to understand how it plays together with the PCI bus
driver, and what values we should expose through it :)

Thanks!
Laszlo
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Brian J. Johnson]

On 06/30/2016 08:07 AM, Laszlo Ersek wrote:

Hi,

does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?

The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.

In my understanding:

* Each PCIe downstream port qualifies as a separate bus / bridge.

* Bridges need IO and MMIO resources for themselves. When a device is
   behind a bridge at boot, the device's IO and MMIO BARs are carved out
   of the bridge's IO and MMIO apertures. For IO, the standard bridge
   aperture size is 4096 (byte-sized) ports.

* If no device behind a bridge needs a specific type of resource (for
   example, no devices behind a bridge have IO BARs), then that kind of
   aperture is not necessary for the bridge either.

   A special case of this is when there are no devices behind a bridge
   at all. Example: PCIe downstream port empty at boot. In this case,
   the bridge needs no resources at all, at boot.

* Different firmwares seem to follow different policies in the last
   case (-> bridge empty at boot). SeaBIOS for example allocates the
   0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
   allocates no apertures for bridges that have no devices behind them
   at boot.

   This appears to be a trade-off: if the bridge remains empty for the
   entire lifetime of the OS, then any IO ports that have been allocated
   in advance will be in vain (wasted). However, if the aperture is not
   allocated at boot, then hotplug under the bridge may not work --
   resources for the device being hot-plugged cannot be allocated from
   nonexistent bridge apertures.

And that's what we are experiencing with OVMF.


Laszlo,

I believe that's an accurate summary of PCIe behavior.



My questions are:

- Is the above behavior of the edk2 PCI bus driver intentional?


I'm not familiar with the history of the code  But any PCI bridge 
which wants to support hotplug needs resources allocated to it at boot 
time.  As you say, that's a trade-off between wasting resources 
(especially I/O ports) and allowing hotplug.  The authors of the code 
must have opted for avoiding waste vs. allowing hotplug.




- Does PCIe hotplug into downstream ports work with any (phys) firmware
   forked from edk2? Brian, Samer, do you guys have experience with this?


Yes, we support it on our UV server line.  Physically, hotplug is only 
possible in expansion chassis designed for that purpose.  (That's not a 
problem with virtual machines, of course.)  But we've had to do quite a 
bit of work on the host bridge driver to allowing scaling to the sizes 
of systems we support (hundreds of sockets and PCIe slots, dozens of TB 
of RAM), including reserving resources for hotplug only in the specific 
cases where it's possible.




- What could be the difference between root ports and downstream ports?
   (Hotplug into root ports seems to work fine.) Are OSes entitled to
   allocate any unused address space (MMIO and IO) right when a device is
   hot-plugged into a root port?


I'm not sure what the OS rules for this are
--

Brian



   "The national budget must be balanced.  The public debt must be
reduced; the arrogance of the authorities must be moderated and
controlled.  Payments to foreign governments must be reduced, if
the nation doesn't want to go bankrupt.  People must again learn
to work, instead of living on public assistance."
  -- Marcus Tullius Cicero, 55 BC
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

Re: [edk2] PCIe hotplug into downstream port

[Andrew Fish]

> On Jun 30, 2016, at 6:07 AM, Laszlo Ersek  wrote:
> 
> Hi,
> 
> does anyone on this list have experience with $SUBJECT, using physical
> UEFI firmware derived from edk2?
> 

Laszlo,

I don't know the edk2 specific answer from memory, but in general as a system 
gets larger it is harder and harder to efficiently over allocate resources. So 
there is usually some mechanism to configure the over allocation of resources 
and identify which ports support hot plug.

Maybe: 
https://github.com/tianocore/edk2/blob/master/MdePkg/Include/Protocol/PciHotPlugInit.h

Thanks,

Andrew Fish

> The problem we're seeing with OVMF is the following: PCIe hotplug works
> with PCIe root ports, but it doesn't work with PCIe downstream ports.
> The error messages reported by both Linux and Windows guests indicate
> "lack of resources" for the device being plugged.
> 
> In my understanding:
> 
> * Each PCIe downstream port qualifies as a separate bus / bridge.
> 
> * Bridges need IO and MMIO resources for themselves. When a device is
>  behind a bridge at boot, the device's IO and MMIO BARs are carved out
>  of the bridge's IO and MMIO apertures. For IO, the standard bridge
>  aperture size is 4096 (byte-sized) ports.
> 
> * If no device behind a bridge needs a specific type of resource (for
>  example, no devices behind a bridge have IO BARs), then that kind of
>  aperture is not necessary for the bridge either.
> 
>  A special case of this is when there are no devices behind a bridge
>  at all. Example: PCIe downstream port empty at boot. In this case,
>  the bridge needs no resources at all, at boot.
> 
> * Different firmwares seem to follow different policies in the last
>  case (-> bridge empty at boot). SeaBIOS for example allocates the
>  0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
>  allocates no apertures for bridges that have no devices behind them
>  at boot.
> 
>  This appears to be a trade-off: if the bridge remains empty for the
>  entire lifetime of the OS, then any IO ports that have been allocated
>  in advance will be in vain (wasted). However, if the aperture is not
>  allocated at boot, then hotplug under the bridge may not work --
>  resources for the device being hot-plugged cannot be allocated from
>  nonexistent bridge apertures.
> 
> And that's what we are experiencing with OVMF.
> 
> My questions are:
> 
> - Is the above behavior of the edk2 PCI bus driver intentional?
> 
> - Does PCIe hotplug into downstream ports work with any (phys) firmware
>  forked from edk2? Brian, Samer, do you guys have experience with this?
> 
> - What could be the difference between root ports and downstream ports?
>  (Hotplug into root ports seems to work fine.) Are OSes entitled to
>  allocate any unused address space (MMIO and IO) right when a device is
>  hot-plugged into a root port?
> 
> Thanks!
> Laszlo
> ___
> edk2-devel mailing list
> edk2-devel@lists.01.org
> https://lists.01.org/mailman/listinfo/edk2-devel

___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel

[edk2] PCIe hotplug into downstream port

[Laszlo Ersek]

Hi,

does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?

The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.

In my understanding:

* Each PCIe downstream port qualifies as a separate bus / bridge.

* Bridges need IO and MMIO resources for themselves. When a device is
  behind a bridge at boot, the device's IO and MMIO BARs are carved out
  of the bridge's IO and MMIO apertures. For IO, the standard bridge
  aperture size is 4096 (byte-sized) ports.

* If no device behind a bridge needs a specific type of resource (for
  example, no devices behind a bridge have IO BARs), then that kind of
  aperture is not necessary for the bridge either.

  A special case of this is when there are no devices behind a bridge
  at all. Example: PCIe downstream port empty at boot. In this case,
  the bridge needs no resources at all, at boot.

* Different firmwares seem to follow different policies in the last
  case (-> bridge empty at boot). SeaBIOS for example allocates the
  0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
  allocates no apertures for bridges that have no devices behind them
  at boot.

  This appears to be a trade-off: if the bridge remains empty for the
  entire lifetime of the OS, then any IO ports that have been allocated
  in advance will be in vain (wasted). However, if the aperture is not
  allocated at boot, then hotplug under the bridge may not work --
  resources for the device being hot-plugged cannot be allocated from
  nonexistent bridge apertures.

And that's what we are experiencing with OVMF.

My questions are:

- Is the above behavior of the edk2 PCI bus driver intentional?

- Does PCIe hotplug into downstream ports work with any (phys) firmware
  forked from edk2? Brian, Samer, do you guys have experience with this?

- What could be the difference between root ports and downstream ports?
  (Hotplug into root ports seems to work fine.) Are OSes entitled to
  allocate any unused address space (MMIO and IO) right when a device is
  hot-plugged into a root port?

Thanks!
Laszlo
___
edk2-devel mailing list
edk2-devel@lists.01.org
https://lists.01.org/mailman/listinfo/edk2-devel