On Thu, Jul 19, 2012 at 04:46:35PM -0300, Eduardo Habkost wrote:
> On Thu, Jul 19, 2012 at 11:28:54AM -0300, Eduardo Habkost wrote:
> > On Thu, Jul 19, 2012 at 12:58:46PM +0300, Gleb Natapov wrote:
> > > On Tue, Jul 17, 2012 at 06:56:30PM -0300, Eduardo Habkost wrote:
> > > > This patch is an attempt to fix the non-continguous-APIC-ID problem
> > > > without the
> > > > FW_CFG_LAPIC_INFO approach I have sent proposed last week.
> > > >
> > > > Basically, this changes Seabios to probe for APIC IDs directly from the
> > > > CPUs on boot, instead of getting it using fw_cfg, store the found APIC
> > > > IDs on a bitmap, and use that information whe building the MADT, SRAT,
> > > > and SSDT ACPI tables.
> > > >
> > > > To do this properly, we have to decide the meaning of "CPU IDs" in the
> > > > QEMU<->Seabios interfaces, too. I see two possible approaches:
> > > >
> > > > 1) Have Seabios and QEMU agree on a a "CPU identifier", that is
> > > > independent from the APIC ID.
> > > > 2) Always use the Initial APIC ID on all communication between QEMU and
> > > > Seabios.
> > > >
> > > We need to be prepared to support more than 255 cpus. With > 255 cpus
> > > comes x2apic and x2apic has 32bit apic ids. HW does not have to support
> > > all of the bits though, but potentially all the bitmasks can grow
> > > prohibitedly large.
> >
> > I see only two solutions:
> >
> > - Specify an interface/convention for QEMU and Seabios agree upon a "CPU
> > identifier" <=> x2apic <=> LAPIC ID mapping for all CPUs.
> > - Specify new NUMA-information and CPU hotplug interfaces (or extend the
> > existing ones) based on x2apic ID, when Seabios start supporting
> > x2apic.
> >
> > I am not particularly inclined towards any of those two solutions. I
> > dislike them equally. :-)
>
> Oh, it is simpler than I have expected. x2APIC specification:
>
> "The local APIC ID is initialized by hardware with a 32 bit ID (x2APIC
> ID). The lowest 8 bits of the x2APIC ID is the legacy local xAPIC ID,
> and is stored in the upper 8 bits of the APIC register for access in
> xAPIC mode."
>
> And the ACPI specification:
>
> "Logical processors with APIC ID values of 255 and greater are required
> to have a Processor Device object and must convey the processor’s APIC
> information to OSPM using the Processor Local X2APIC structure. Logical
> processors with APIC ID values less than 255 must use the Processor
> Local APIC structure to convey their APIC information to OSPM."
>
> That means the x2APIC ID and the xAPIC ID are interchangeable, for
> values <= 255. That means the QEMU<=>Seabios communication can be safely
> based on "APIC IDs" without any ambiguity.
>
Yes for <= 255 they interchangeable. That's why we can add +x2apic to
our cpu models without changes to the BIOS.
> The CPU hotplug interface is a bit of a problem because it is based on a
> 256-bit bitmap. But on the day it gets extended to support more than 256
> CPUs, it can safely be based on APIC IDs and still keep compatibility
> with systems without x2APIC.
The bitmap will have to be extended if we will go beyond 256 cpus. Using
apic-id to index the bitmap means that the size of the bitmap is a
function of max apic-id we want to support, not max number of cpus.
>
> So, now I am strongly inclined towards the second option from the list
> above: just use APIC IDs everywhere to identify CPUs when QEMU and
> Seabios communicate with each other, and QEMU can completely ignore the
> "Processor ID" used by Seabios.
I agree with making "Processor ID" Seabios internal thing.
>
> >
> > Note that I am more worried about the QEMU<->Seabios interfaces. The
> > APIC ID bitmap on smp.c, for example, is just an implementation detail:
> > if we make Seabios support x2apic, that code can be changed to use a
> > different data structure instead.
> >
> [...]
> > > > To try to make things less likely to break on the common, non-hotplug
> > > > case, this patch makes the Processor IDs be chosen this way:
> > > >
> > > > - The CPUs present on boot get contiguous Processor IDs (even if the
> > > > APIC IDs are not contiguous);
> > > > - The remaining Processor declarations are going to associated to the
> > > > remaining APIC IDs (immediately after the last present APIC ID),
> > > > sequentially. This means that hotplugged CPUs may not get contiguous
> > > > Processor declarations if their APIC IDs are not contiguous.
> > > >
> > > I am curious what will happen if cpu will be hot plugged, than hibernate
> > > and resume is done. After resume hot plugged cpu will have different
> > > Processor ID in ACPI. This may or may not be a problem.
> >
> > True. Keeping those tables stable after hotplug and hibernate may be a
> > challenge.
> >
> > Maybe it would be easier to just leave holes on the MADT and SSDT tables
> > (making APIC ID and Processor ID always match), and hope no OS will be
> > confused by the holes.
>
> I am inclined to try this approach first (keep APIC ID == ACPI Processor
> ID), to keep things simple in Seabios. I am hoping no OS will have
> problems with the holes in the list of enabled Processor IDs.
>
They shouldn't.
--
Gleb.
