On 06/30/2016 11:47 AM, Laszlo Ersek wrote:
On 06/30/16 18:39, Marcel Apfelbaum wrote:
On 06/30/2016 07:21 PM, Marcel Apfelbaum wrote:
On 06/30/2016 04:07 PM, Laszlo Ersek wrote:
Hi,
does anyone on this list have experience with $SUBJECT, using physical
UEFI firmware derived from edk2?
The problem we're seeing with OVMF is the following: PCIe hotplug works
with PCIe root ports, but it doesn't work with PCIe downstream ports.
The error messages reported by both Linux and Windows guests indicate
"lack of resources" for the device being plugged.
In my understanding:
* Each PCIe downstream port qualifies as a separate bus / bridge.
* Bridges need IO and MMIO resources for themselves. When a device is
behind a bridge at boot, the device's IO and MMIO BARs are carved out
of the bridge's IO and MMIO apertures. For IO, the standard bridge
aperture size is 4096 (byte-sized) ports.
* If no device behind a bridge needs a specific type of resource (for
example, no devices behind a bridge have IO BARs), then that kind of
aperture is not necessary for the bridge either.
A special case of this is when there are no devices behind a bridge
at all. Example: PCIe downstream port empty at boot. In this case,
the bridge needs no resources at all, at boot.
* Different firmwares seem to follow different policies in the last
case (-> bridge empty at boot). SeaBIOS for example allocates the
0x1000 IO ports nonetheless. The PCI bus driver in edk2 does not: it
allocates no apertures for bridges that have no devices behind them
at boot.
This appears to be a trade-off: if the bridge remains empty for the
entire lifetime of the OS, then any IO ports that have been allocated
in advance will be in vain (wasted). However, if the aperture is not
allocated at boot, then hotplug under the bridge may not work --
resources for the device being hot-plugged cannot be allocated from
nonexistent bridge apertures.
Hi Laszlo,
I fully agree with your assessment.
From the firmware point of view it seems like a waste to allocate an
IO/MEM range for
an unused PCI/PCIe-bridge indeed. Especially IO.
But what about MEM?
Let's have a look to this special case: PCIe downstream ports on
64-bit systems.
1. PCIe spec requires devices to work correctly even without enabling
their IO BARs.
2. 64-bit Memory space is pretty large, decent CPUs support 46-bit
addresses, so even if the system uses Terabytes of RAM
and has dozens of switches we still have enough unused MEM space.
My point is we can skip assigning IO for PCIe ports but we can assign
a 4K MEM range in 64-bit memory for them
without being too greedy.
I meant 2M MEM range, sorry. 4k is the min range for IO, of course.
One thing to watch out for is prefetchable vs. non-prefetchable MEM
ranges. On Intel chipsets, the high (64-bit) MMIO range is always
prefetchable, which mainly makes it useful for frame buffers,
accelerator cards with RAM mapped to MMIO in the host, and the like.
Only the low (32-bit) MMIO space is non-prefetchable, and suitable for
registers and things which need precise ordering of loads and stores.
So you can demote a prefetchable BAR into non-prefetchable MMIO space if
necessary, but not the other way around. Which is a shame, because the
32-bit MMIO space is much more limited in size than the 64-bit space.
I've just been looking at the values reserved by SeaBIOS [src/fw/pciinit.c]:
#define PCI_BRIDGE_MEM_MIN (1<<21) // 2M == hugepage size
#define PCI_BRIDGE_IO_MIN 0x1000 // mandated by pci bridge spec
2MB sounds perfectly fine for MEM, especially given that OVMF supports a
64-bit MMIO aperture (32GB in size by default).
We discussed the IO size earlier -- I think 4K is overkill (despite
being required by the standard). We considered half K, 1K, 2K earlier...
I think I'll take a stab at half K first.
It depends on what your hardware (physical or virtual) supports. A
standards-compliant bridge may not be able to map I/O ports to targets
on less than a 4k boundary. Yes, 4k is ridiculous overkill, but it may
be your only choice, depending on the details of the h/w and/or VMM.
We've had to add special cases in BIOS for certain cards we support
which request IO space, but don't actually use it. We can safely skip
allocating IO ports to those cards, saving significant space in the
overall IO port map. We've worked with the card vendors on cleaning up
their PCIe advertisements and eliminating the use of IO ports, but it
takes time.
--
Brian
--------------------------------------------------------------------
"My software never has bugs. It just develops random features."
-- Quoted by Khan Tran
_______________________________________________
edk2-devel mailing list
[email protected]
https://lists.01.org/mailman/listinfo/edk2-devel
