Hi Edgar, > On 20 May 2025, at 10:04, Edgar E. Iglesias <edgar.igles...@amd.com> wrote: > > Hi all, > > Following up on the ARM virtio-pci series I posted a while back ago. > > There have been some concerns around the delayed and silent apperance of > devices on the ECAM area. The spec is not super clear wether this is OK or > not but I'm providing some references to the PCI specs and to some real cases > where this is used for FPGAs. > > There are two options how to implement virtio-pci that we've discussed: > 1. VPCI + IOREQ > 2. IOREQ only > > There are pros and cons with both. For example with #1, has the benefit that > we would only have a single PCIe RC (in Xen) and we could emulate a hotplug > capable expansion port with a standard way to notify when PCI devices plug in. > Approach #2 has the benefit that there is (almost) no additional complexity > or code added to Xen, almost everything lives outside. > IMO, both options have merit and both could co-exist. > > For dynamic xl flows, options #2 already works without modifications to Xen. > Users need to pass the correct command-line options to QEMU and a device-tree > fragment with the pci-generic-ecam-host device. > > For static dom0less flows, we can do the same as for xl flows but we have the > additional problem of domU's PCI bus enumeration racing with QEMU. > On x86, when domU's access a memory range that has not yet got IOREQ's > connected to it, the accesses succeeds with reads returning 0xFFFFFFFF and > writes ignored. This makes it easy for guests to wait for IOREQ devices to > pop up since guests will find an empty bus and can initiate a rescan later > when QEMU attaches. On ARM, we trap on these accesses. > > If we on ARM add support for MMIO background regions with a default read > value, > i.e mmio handlers that have lower priority than IOREQs and that are > read-const + writes-ignored, we could support the same flow on ARM. > This may be generally useful for other devices as well (e.g virtio-mmio or > something else). We could also use this to defer PCI enumeration. > > So the next versions of this series I was thinking to remove the PCI specifics > and instead add FDT bindings to ARM dom0less enabling setup of user > configurable (by address-range and read-value) background mmio regions. > Xen would then support option #2 without any PCI specifics added. > > Thoughts?
We discussed that together last week and I think this is a good approach as it prevents having some "workaround" code for PCI not following the Virtio spec and could also fulfil some other use cases by giving a solution to emulate some IO registers for a guest with a fixed value. Cheers Bertrand > > Cheers, > Edgar > > # References to spec > > PCI express base specification: > 7.5.1.1.1 Vendor ID Register (Offset 00h) > The Vendor ID register is HwInit and the value in this register identifies > the manufacturer of the Function. In keeping with > PCI-SIG procedures, valid vendor identifiers must be allocated by the PCI-SIG > to ensure uniqueness. Each vendor must > have at least one Vendor ID. It is recommended that software read the Vendor > ID register to determine if a Function is > present, where a value of FFFFh indicates that no Function is present. > > PCI Firmware Specification: > 3.5 Device State at Firmware/Operating System Handoff > Page 34: > The operating system is required to configure PCI subsystems: > During hotplug > For devices that take too long to come out of reset > PCI-to-PCI bridges that are at levels below what firmware is designed to > configure > > Page 36: > Note: The operating system does not have to walk all buses during boot. The > kernel can > automatically configure devices on request; i.e., an event can cause a scan > of I/O on demand. > > FPGA's can be programmed at runtime and appear on the ECAM bus silently. > An PCI rescan needs to be triggered for the OS to discover the device: > Intel FPGAs: > https://www.intel.com/content/www/us/en/docs/programmable/683190/1-3-1/how-to-rescan-bus-and-re-enable-aer.html >
