Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2019-01-04 Thread Laine Stump 

On 1/3/19 5:22 AM, Riccardo Ravaioli wrote:
On Thu, 20 Dec 2018 at 15:39, Laine Stump > wrote:


I think you're right. Each bus requires some amount of IO space, and I
thought I recalled someone saying that all of the available IO space is
exhausted after 7 or 8 buses. [...]


Laine,

Do you have by any chance a link to a page explaining this in more details?
Thanks again! :)


No, sorry. There are mentions of it in bugzilla records (e.g. 
https://bugzilla.redhat.com/show_bug.cgi?id=106 ) but all the info I 
have is just recalled from email and irc conversations over the last 3-4 
years. Basically the amount of IO address space is limited, and SeaBIOS 
allocates some minimum-sized chunk of that for each PCI controller that 
is probed and "seems to need IO address space for its devices". After 8 
or so controllers, all the space is used up.


You can avoid the IO address space limit if you're using a PCI Express 
based machinetype, *and* all PCIe devices, but you'll run up against 
different limits that need to be worked around in a different way - PCIe 
devices are required to be usable with no IO address space (and so QEMU 
specifically creates the PCIe controllers beyond pcie-root without any). 
But each PCIe controller only has a single slot (with 8 functions), each 
consumes a "bus number" and the bus number is an 8-bit value, so you're 
limited to 256 total (x8 if you don't need hotplug and don't mind 
manually assigning addresses).


___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2019-01-03 Thread Riccardo Ravaioli 
On Thu, 20 Dec 2018 at 15:39, Laine Stump  wrote:

> I think you're right. Each bus requires some amount of IO space, and I
> thought I recalled someone saying that all of the available IO space is
> exhausted after 7 or 8 buses. [...]
>

Laine,

Do you have by any chance a link to a page explaining this in more details?
Thanks again! :)

Riccardo
___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2018-12-20 Thread Riccardo Ravaioli 
On Thu, 20 Dec 2018 at 15:39, Laine Stump  wrote:

> [...]
> I think you're right. Each bus requires some amount of IO space, and I
> thought I recalled someone saying that all of the available IO space is
> exhausted after 7 or 8 buses. This was in relation to PCIe, where each
> root port is a bus, and can potentially take up IO space, so possibly in
> that context they were talking about the buses *after* the root bus and
> pcie-pci-bridge, which would bring us back to the same number you're
> getting.
>
> For PCIe our solution was to turn off IO space usage on the
> pcie-root-ports, but you can't do that for conventional PCI buses, since
> the devices might actually need IO space to function properly and the
> standard requires that you provide it.
>

Ok, that makes sense.


> > The real question though is why you need to create so many PCI buses.
> > Each bus can do 31 devices.  Do you really need to have more than 279
> > devices in your VM ?
>
> And if you do need more than 279 devices, do they all need to be
> hot-pluggable? If not, then you can put up to 8 devices on each slot
> (functions 0 - 7).
>

True. I'll use the function field too, then.

Thanks a lot!

Riccardo
___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2018-12-20 Thread Riccardo Ravaioli 
On Thu, 20 Dec 2018 at 15:20, Daniel P. Berrangé 
wrote:

> [...]
>
> I guess the hang is that you hit some limit in PCI buses.
>
> The real question though is why you need to create so many PCI buses.
> Each bus can do 31 devices.  Do you really need to have more than 279
> devices in your VM ?
>

Ok, I see. Of course I don't really need that many devices, I was just
exploring the available ranges in a PCI address :)

Riccardo
___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2018-12-20 Thread Laine Stump 

On 12/20/18 9:20 AM, Daniel P. Berrangé wrote:

On Thu, Dec 20, 2018 at 03:15:34PM +0100, Riccardo Ravaioli wrote:

Hi,

My goal is to assign PCI addresses to a number of devices (network
interfaces, disks and PCI devices in PCI-passthrough) without delegating to
libvirt the generation of those values. This should give me more control
and for sure more predictability over the hardware configuration of a
virtual machine and consequently the name of the interfaces in it.  I'm
using libvirt 4.3.0 to create qemu/KVM virtual machines running Linux
(Debian Stretch).

So, for every device of the type mentioned above, I add this line:
,
... with values from 00 to ff in the bus field, and from 00 to 1f in the
slot field, as described in the documentation.

Long story short, I noticed that as soon as I assign values > 0x09 to the
bus field, the serial console hangs indefinitely, in both Debian and
Ubuntu. The VM seems to be started correctly and its state is "running"; in
the XML file created by libvirt, I see all controllers from 0 the largest
bus value I assigned, so everything from that side seems ok.


I guess the hang is that you hit some limit in PCI buses.


I think you're right. Each bus requires some amount of IO space, and I 
thought I recalled someone saying that all of the available IO space is 
exhausted after 7 or 8 buses. This was in relation to PCIe, where each 
root port is a bus, and can potentially take up IO space, so possibly in 
that context they were talking about the buses *after* the root bus and 
pcie-pci-bridge, which would bring us back to the same number you're 
getting.


For PCIe our solution was to turn off IO space usage on the 
pcie-root-ports, but you can't do that for conventional PCI buses, since 
the devices might actually need IO space to function properly and the 
standard requires that you provide it.





The real question though is why you need to create so many PCI buses.
Each bus can do 31 devices.  Do you really need to have more than 279
devices in your VM ?


And if you do need more than 279 devices, do they all need to be 
hot-pluggable? If not, then you can put up to 8 devices on each slot 
(functions 0 - 7).




Regards,
Daniel



___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users

Re: [libvirt-users] assigning PCI addresses with bus > 0x09

2018-12-20 Thread Daniel P . Berrangé 
On Thu, Dec 20, 2018 at 03:15:34PM +0100, Riccardo Ravaioli wrote:
> Hi,
> 
> My goal is to assign PCI addresses to a number of devices (network
> interfaces, disks and PCI devices in PCI-passthrough) without delegating to
> libvirt the generation of those values. This should give me more control
> and for sure more predictability over the hardware configuration of a
> virtual machine and consequently the name of the interfaces in it.  I'm
> using libvirt 4.3.0 to create qemu/KVM virtual machines running Linux
> (Debian Stretch).
> 
> So, for every device of the type mentioned above, I add this line:
> ,
> ... with values from 00 to ff in the bus field, and from 00 to 1f in the
> slot field, as described in the documentation.
> 
> Long story short, I noticed that as soon as I assign values > 0x09 to the
> bus field, the serial console hangs indefinitely, in both Debian and
> Ubuntu. The VM seems to be started correctly and its state is "running"; in
> the XML file created by libvirt, I see all controllers from 0 the largest
> bus value I assigned, so everything from that side seems ok.

I guess the hang is that you hit some limit in PCI buses.

The real question though is why you need to create so many PCI buses.
Each bus can do 31 devices.  Do you really need to have more than 279
devices in your VM ?

Regards,
Daniel
-- 
|: https://berrange.com  -o-https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o-https://fstop138.berrange.com :|
|: https://entangle-photo.org-o-https://www.instagram.com/dberrange :|

___
libvirt-users mailing list
libvirt-users@redhat.com
https://www.redhat.com/mailman/listinfo/libvirt-users