Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[David Gibson]

On Wed, Dec 16, 2015 at 06:22:20PM +0100, Igor Mammedov wrote:
> On Wed, 16 Dec 2015 16:57:54 +0100
> Andreas Färber  wrote:
> 
> > Am 16.12.2015 um 16:44 schrieb Igor Mammedov:
> > > On Wed, 16 Dec 2015 16:19:06 +0100
> > > Andreas Färber  wrote:
> > > 
> > >> Am 10.12.2015 um 07:15 schrieb Bharata B Rao:
> > >>> CPU hotplug granularity
> > >>> ---
> > >>> CPU hotplug will now be done in cpu-core device granularity.
> > >>
> > >> Nack.
> > >>
> > >>> Are there archs that would need thread level CPU addition ?
> > >>
> > >> Yes, s390. And for x86 people called for socket level.
> > > socket level hotplug would be the last resort if we can't agree
> > > on thread level one. As it would break existing setups where
> > > user can hotplug 1 core, and  I'd like to avoid it if it is
> > > possible.
> > 
> > We still need to keep cpu-add for backwards compatibility, so I am
> > discussing solely the new device_add interface. My previous x86 series
> > went to severe hacks trying to keep cpu-add working with
> > sockets
> if possible, it would be better to make cpu-add to use device_add
> internally.
> 
> > 
> > Attendees in Seattle said that thread-level hot-plug were dangerous
> > for Linux guests due to assumptions in the (guest's) scheduler
> > breaking for any incompletely filled cores or sockets. No one present
> There is not such thing as cpu hotplug at socket level in x86 linux so far.
> CPUs are plugged at logical(thread) cpu level, one at a time.
> And ACPI spec does the same (describes logical CPUs) and hotplug
> notification in guest handled per one logical cpu at a time.

I don't think that precludes handling hotplug at socket level in
qemu.  The user <-> qemu interaction can work on the socket level,
then the qemu <-> guest interaction on the thread level, iterating
through the threads in the socket.

Problems arise when the qemu <-> guest protocol is at a coarser
granularity than the user <-> qemu protocol, rather than the other way
around, AFAICT.

This is the problem we have with the existing interfaces for Power -
there the qemu <-> guest protocol specified by the platform has no way
of representing a single thread hotplug, it's always a core at a time
(as a paravirt platform, there's not really a useful difference
between cores and sockets).

-- 
David Gibson| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au  | minimalist, thank you.  NOT _the_ _other_
| _way_ _around_!
http://www.ozlabs.org/~dgibson


signature.asc
Description: PGP signature

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Fri, 1 Jan 2016 09:17:48 +0530
Bharata B Rao  wrote:

> On Wed, Dec 16, 2015 at 04:46:37PM +0100, Andreas Färber wrote:
> > Am 10.12.2015 um 13:35 schrieb Igor Mammedov:  
> > > wrt CLI can't we do something like this?
> > > 
> > > -device some-cpu-model,socket=x[,core=y[,thread=z]]  
> > 
> > That's problematic and where my x86 remodeling got stuck. It works fine
> > (more or less) to model sockets, cores and hyperthreads for -smp, but
> > doing it dynamically did not work well. How do you determine the
> > instance size a socket with N cores and M threads needs? Allocations in
> > instance_init are to be avoided with a view to hot-plug. So either we
> > have a fully determined socket object or we need to wire individual
> > objects on the command line. The latter has bad implications for
> > atomicity and thus hot-unplug. That leaves us with dynamic properties
> > doing allocations and reporting it via Error**, something I never
> > finished and could use reviewers and contributors.
> > 
> > Anthony's old suggestion had been to use real socket product names like
> > Xeon-E5-4242 to get a 6-core, dual-thread socket, without parameters -
> > unfortunately I still don't see an easy way to define such a thing today
> > with the flexibility users will undoubtedly want.
> > 
> > And since the question came up how to detect this, what you guys seem to
> > keep forgetting is that somewhere there also needs to be a matching
> > link<> property that determines what can be plugged, i.e. QMP qom-list.
> > link<>s are the QOM equivalent to qdev's buses. The object itself needs
> > to live in /machine/peripheral or /machine/peripheral-anon
> > (/machine/unattached is supposed to go away after the QOM conversion is
> > done!) and in a machine-specific place there will be a
> > /machine/cpu-socket[0] -> /machine/peripheral-anon/device[42]
> > link property. It might just as well be
> > /machine/daughterboard-x/cpu-core[2] -> /machine/peripheral/cpu0.
> > (Gentle reminder of the s390 ipi modeling discussion that never came to
> > any conclusion iirc.)
> > 
> > Note that I have not read this patch series yet, just some of the
> > alarming review comments.  
> 
> It has been more than an year since I posted the initial version of
> PowerPC sPAPR CPU hotplug patchset. I guess x86 CPU hotplug patchset
> existed even before that. Now we have patches for s390 CPU hotplug
> also on the list. Given this situation, will it be agreeable and
> feasible to follow Igor's suggestion and de-link the QOM part from the
> actual CPU hotplug work ? May be we can get these patchsets into 2.6 and
> work on QOM links subsequently ?
how about doing it in 2 series in parallel,
 1st: a target specific cpu hotplug
 2nd: QMP interface that would suite management needs to enumerate
  existing/possible CPU objects at the granularity which targets
  support from -device/device_add aspect
  (i.e. focuses only on hotplug POV)


> Regards,
> Bharata.
> 
> 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Bharata B Rao]

On Wed, Dec 16, 2015 at 04:46:37PM +0100, Andreas Färber wrote:
> Am 10.12.2015 um 13:35 schrieb Igor Mammedov:
> > wrt CLI can't we do something like this?
> > 
> > -device some-cpu-model,socket=x[,core=y[,thread=z]]
> 
> That's problematic and where my x86 remodeling got stuck. It works fine
> (more or less) to model sockets, cores and hyperthreads for -smp, but
> doing it dynamically did not work well. How do you determine the
> instance size a socket with N cores and M threads needs? Allocations in
> instance_init are to be avoided with a view to hot-plug. So either we
> have a fully determined socket object or we need to wire individual
> objects on the command line. The latter has bad implications for
> atomicity and thus hot-unplug. That leaves us with dynamic properties
> doing allocations and reporting it via Error**, something I never
> finished and could use reviewers and contributors.
> 
> Anthony's old suggestion had been to use real socket product names like
> Xeon-E5-4242 to get a 6-core, dual-thread socket, without parameters -
> unfortunately I still don't see an easy way to define such a thing today
> with the flexibility users will undoubtedly want.
> 
> And since the question came up how to detect this, what you guys seem to
> keep forgetting is that somewhere there also needs to be a matching
> link<> property that determines what can be plugged, i.e. QMP qom-list.
> link<>s are the QOM equivalent to qdev's buses. The object itself needs
> to live in /machine/peripheral or /machine/peripheral-anon
> (/machine/unattached is supposed to go away after the QOM conversion is
> done!) and in a machine-specific place there will be a
> /machine/cpu-socket[0] -> /machine/peripheral-anon/device[42]
> link property. It might just as well be
> /machine/daughterboard-x/cpu-core[2] -> /machine/peripheral/cpu0.
> (Gentle reminder of the s390 ipi modeling discussion that never came to
> any conclusion iirc.)
> 
> Note that I have not read this patch series yet, just some of the
> alarming review comments.

It has been more than an year since I posted the initial version of
PowerPC sPAPR CPU hotplug patchset. I guess x86 CPU hotplug patchset
existed even before that. Now we have patches for s390 CPU hotplug
also on the list. Given this situation, will it be agreeable and
feasible to follow Igor's suggestion and de-link the QOM part from the
actual CPU hotplug work ? May be we can get these patchsets into 2.6 and
work on QOM links subsequently ?

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Thu, 24 Dec 2015 09:59:53 +0800
Zhu Guihua  wrote:

> 
> On 12/17/2015 05:58 AM, Igor Mammedov wrote:
> > On Wed, 16 Dec 2015 16:46:37 +0100
> > Andreas Färber  wrote:
> >
> >> Am 10.12.2015 um 13:35 schrieb Igor Mammedov:
> >>> wrt CLI can't we do something like this?
> >>>
> >>> -device some-cpu-model,socket=x[,core=y[,thread=z]]
> >> That's problematic and where my x86 remodeling got stuck. It works
> >> fine (more or less) to model sockets, cores and hyperthreads for
> >> -smp, but doing it dynamically did not work well. How do you
> >> determine the instance size a socket with N cores and M threads
> >> needs?
> > -smp defines necessary topology, all one need is to find out
> > instance size of a core or thread object, it would be probably x86
> > specific but it's doable, the only thing in current X86 thread
> > needs to fix is to reserve space for largest APIC type
> > and replace object_new with object_initialize_with_type.
> >
> > However if I look from x86 point of view, there isn't need to model
> > sockets nor cores. The threads are sufficient for QEMU needs.
> > Dummy sockets/cores are just complicating implementation.
> >
> > What I'm advocating for is let archs to decide if they should create
> > CPUs per socket, core or thread.
> >
> > And for x86 do this at thread level, that way we keep compatibility
> > with cpu-add, but will also allow which cpu thread to plug with
> >   'node=n,socket=x,core=y,thread=z'.
> >
> > Another point in favor of thread granularity for x86 is that competing
> > hyper-visors are doing that at thread level, QEMU would be worse off
> > in feature parity if minimal hotplug unit would be a socket.
> >
> > That also has benefit of being very flexible and would also suit
> > engineering audience of QEMU, allowing them to build CPUs from
> > config instead of hardcoding it in code and playing heterogeneous
> > configurations.
> >
> > Options 'node=n,socket=x,core=y,thread=z' are just a SMP specific
> > path, defining where CPU should attached, it could be a QOM path
> > in the future when we arrive there and have a stable QOM tree.
> 
> This means we will cancel the current apic_id in command line?
> And we have already realized the options 'node=n' one year ago.
There isn't apic_id option in command line so far and the reason for it
is that it's complex to calculate and depends on CPU type beside of
being x86 specific only.

on contrary
 -device some-cpu-model,node=n,socket=x,core=y,thread=z
provides all necessary information for QEMU to calculate APIC ID
while being relatively arch neutral.
Then each arch can require a full set of these options or a subset
to be provided at -device/device_add so it could plug CPU at board
level when hotplug_handler_plug() is called for it.


> 
> Thanks,
> Zhu
> 
> >> Allocations in instance_init are to be avoided with a view to
> >> hot-plug.
> >> So either we have a fully determined socket object or we
> >> need to wire individual objects on the command line. The latter has
> >> bad implications for atomicity and thus hot-unplug. That leaves us
> > what are these bad implication and how they affect unplug?
> >
> > If for example x86 CPU thread is fixed to embed child APIC then
> > to avoid allocations as much as possible or fail gracefully there
> > is 2 options :
> >   1: like you've said, reserve all needed space at startup, i.e.
> >  pre-create empty sockets
> >   2: fail gracefully in qdev_device_add() if allocation is not possible
> >
> > for #2 it's not enough to avoid allocations in instance_init()
> > we also must teach qdev_device_add() to get the size of to be created
> > object and replace
> > object_new() with malloc() + object_initialize_with_type(),
> > that way it's possible to fail allocation gracefully and report error.
> >
> > Doing that would benefit not only CPUs but every device_add capable
> > Device and is sufficient for hotplug purposes without overhead of
> > reserving space for every possible hotplugged device at startup (which
> > is impossible anyway in generic)
> > So I'd go for #2 sane device_add impl. vs #1 preallocated objects one
> >   
> >> with dynamic properties doing allocations and reporting it via
> >> Error**, something I never finished and could use reviewers and
> >> contributors.
> > most of dynamic properties are static, looks like what QOM needs
> > is really static properties so we don't misuse the former and probably
> > a way to reserve space for declared number of dynamic ones to avoid
> > allocations in instance_initialize().
> >
> >> Anthony's old suggestion had been to use real socket product names
> >> like Xeon-E5-4242 to get a 6-core, dual-thread socket, without
> >> parameters - unfortunately I still don't see an easy way to define
> >> such a thing today with the flexibility users will undoubtedly want.
> > I don't see it either and for me it is much harder to remember
> > what Xeon-E5-4242 is while it's much easier to say:

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Zhu Guihua]

On 12/17/2015 05:58 AM, Igor Mammedov wrote:

On Wed, 16 Dec 2015 16:46:37 +0100
Andreas Färber  wrote:


Am 10.12.2015 um 13:35 schrieb Igor Mammedov:

wrt CLI can't we do something like this?

-device some-cpu-model,socket=x[,core=y[,thread=z]]

That's problematic and where my x86 remodeling got stuck. It works
fine (more or less) to model sockets, cores and hyperthreads for
-smp, but doing it dynamically did not work well. How do you
determine the instance size a socket with N cores and M threads
needs?

-smp defines necessary topology, all one need is to find out
instance size of a core or thread object, it would be probably x86
specific but it's doable, the only thing in current X86 thread
needs to fix is to reserve space for largest APIC type
and replace object_new with object_initialize_with_type.

However if I look from x86 point of view, there isn't need to model
sockets nor cores. The threads are sufficient for QEMU needs.
Dummy sockets/cores are just complicating implementation.

What I'm advocating for is let archs to decide if they should create
CPUs per socket, core or thread.

And for x86 do this at thread level, that way we keep compatibility
with cpu-add, but will also allow which cpu thread to plug with
  'node=n,socket=x,core=y,thread=z'.

Another point in favor of thread granularity for x86 is that competing
hyper-visors are doing that at thread level, QEMU would be worse off
in feature parity if minimal hotplug unit would be a socket.

That also has benefit of being very flexible and would also suit
engineering audience of QEMU, allowing them to build CPUs from
config instead of hardcoding it in code and playing heterogeneous
configurations.

Options 'node=n,socket=x,core=y,thread=z' are just a SMP specific
path, defining where CPU should attached, it could be a QOM path
in the future when we arrive there and have a stable QOM tree.


This means we will cancel the current apic_id in command line?
And we have already realized the options 'node=n' one year ago.

Thanks,
Zhu


Allocations in instance_init are to be avoided with a view to
hot-plug.
So either we have a fully determined socket object or we
need to wire individual objects on the command line. The latter has
bad implications for atomicity and thus hot-unplug. That leaves us

what are these bad implication and how they affect unplug?

If for example x86 CPU thread is fixed to embed child APIC then
to avoid allocations as much as possible or fail gracefully there
is 2 options :
  1: like you've said, reserve all needed space at startup, i.e.
 pre-create empty sockets
  2: fail gracefully in qdev_device_add() if allocation is not possible

for #2 it's not enough to avoid allocations in instance_init()
we also must teach qdev_device_add() to get the size of to be created
object and replace
object_new() with malloc() + object_initialize_with_type(),
that way it's possible to fail allocation gracefully and report error.

Doing that would benefit not only CPUs but every device_add capable
Device and is sufficient for hotplug purposes without overhead of
reserving space for every possible hotplugged device at startup (which
is impossible anyway in generic)
So I'd go for #2 sane device_add impl. vs #1 preallocated objects one
  

with dynamic properties doing allocations and reporting it via
Error**, something I never finished and could use reviewers and
contributors.

most of dynamic properties are static, looks like what QOM needs
is really static properties so we don't misuse the former and probably
a way to reserve space for declared number of dynamic ones to avoid
allocations in instance_initialize().


Anthony's old suggestion had been to use real socket product names
like Xeon-E5-4242 to get a 6-core, dual-thread socket, without
parameters - unfortunately I still don't see an easy way to define
such a thing today with the flexibility users will undoubtedly want.

I don't see it either and for me it is much harder to remember
what Xeon-E5-4242 is while it's much easier to say:
I want N [cpu-foo] threads
which in SMP world could be expressed via add N tread objects at
specified locations
device_add cpu-foo, with optional node=n,socket=x,core=y,thread=z
allows to do it.
And well for x86 there lots of these Xeon-foo/whatever-foo codenames,
which would be nightmare to maintain.


And since the question came up how to detect this, what you guys seem
to keep forgetting is that somewhere there also needs to be a matching
link<> property that determines what can be plugged, i.e. QMP
qom-list. link<>s are the QOM equivalent to qdev's buses. The object
itself needs to live in /machine/peripheral
or /machine/peripheral-anon (/machine/unattached is supposed to go
away after the QOM conversion is done!) and in a machine-specific
place there will be a /machine/cpu-socket[0]
-> /machine/peripheral-anon/device[42] link
property. It might just as well
be /machine/daughterboard-x/cpu-core[2] -> 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Peter Krempa]

On Wed, Dec 16, 2015 at 16:11:08 +0100, Igor Mammedov wrote:
> On Fri, 11 Dec 2015 09:27:57 +0530
> Bharata B Rao  wrote:
> 
> > On Thu, Dec 10, 2015 at 01:35:05PM +0100, Igor Mammedov wrote:
> > > On Thu, 10 Dec 2015 11:45:35 +0530
> > > Bharata B Rao  wrote:

[...]

> > > For initial conversion of x86-cpus to device-add we could do pretty
> > > much the same like we do now, where cpu devices will appear under:
> > > /machine (pc-i440fx-2.5-machine)
> > >   /unattached (container)
> > > /device[x] (qemu64-x86_64-cpu)
> > > 
> > > since we don't have to maintain/model dummy socket/core objects.
> > > 
> > > PowerPC could do the similar only at core level since it has
> > > need for modeling core objects.
> > > 
> > > It doesn't change anything wrt current introspection state, since
> > > cpus could be still found by mgmt tools that parse QOM tree.
> > > 
> > > We probably should split 2 conflicting goals we are trying to meet
> > > here,
> > > 
> > >  1. make device-add/dell work with cpus /
> > >  drop support for cpu-add in favor of device_add 
> > > 
> > >  2. how to model QOM tree view for CPUs in arch independent manner
> > > to make mgmt layer life easier.
> > > 
> > > and work on them independently instead of arguing for years,
> > > that would allow us to make progress in #1 while still thinking
> > > about how to do #2 the right way if we really need it.
> > 
> > Makes sense, s390 developer also recommends the same. Given that we
> > have CPU hotplug patchsets from x86, PowerPC and s390 all
> > implementing device_add semantics pending on the list, can we hope to
> > get them merged for QEMU-2.6 ?
> > 
> > So as seen below, the device is either "cpu_model-cpu_type" or just
> > "cpu_type".
> generic device_add isn't able to deal with 'cpu_model' stuff, so
> it should be concrete 'cpu_type'.
> Question is if libvirt can get a list of supported CPU types. 
> 
> > 
> > -device POWER8-powerpc64-cpu (pseries)
> > -device qemu64-x86_64-cpu (pc)
> > -device s390-cpu (s390)
> > 
> > Is this going to be the final acceptable semantics ? Would libvirt be
> > able to work with this different CPU device names for different
> > guests ?
> CCing Peter to check if libvirt could do it nad if his is ok with
> proposed device_add semantics as in the it's he who will deal with it
> on libvirt side.

Well, this depends entirely on the implementation and the variety of the
cpu device models. Libvirt requires that the cpu model for a given
arch/machine type/whatever can be inferred either completely offline or
via monitor commands that are available when qemu is started with the
'none' machine type. This is required as we query capabilities of a qemu
binary beforehand and then use the cached data to create the command
line. Running a qemu process just to query a cpu model is not acceptable
as it adds significant overhead.

Peter


signature.asc
Description: Digital signature

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Wed, 16 Dec 2015 16:57:54 +0100
Andreas Färber  wrote:

> Am 16.12.2015 um 16:44 schrieb Igor Mammedov:
> > On Wed, 16 Dec 2015 16:19:06 +0100
> > Andreas Färber  wrote:
> > 
> >> Am 10.12.2015 um 07:15 schrieb Bharata B Rao:
> >>> CPU hotplug granularity
> >>> ---
> >>> CPU hotplug will now be done in cpu-core device granularity.
> >>
> >> Nack.
> >>
> >>> Are there archs that would need thread level CPU addition ?
> >>
> >> Yes, s390. And for x86 people called for socket level.
> > socket level hotplug would be the last resort if we can't agree
> > on thread level one. As it would break existing setups where
> > user can hotplug 1 core, and  I'd like to avoid it if it is
> > possible.
> 
> We still need to keep cpu-add for backwards compatibility, so I am
> discussing solely the new device_add interface. My previous x86 series
> went to severe hacks trying to keep cpu-add working with
> sockets
if possible, it would be better to make cpu-add to use device_add
internally.

> 
> Attendees in Seattle said that thread-level hot-plug were dangerous
> for Linux guests due to assumptions in the (guest's) scheduler
> breaking for any incompletely filled cores or sockets. No one present
There is not such thing as cpu hotplug at socket level in x86 linux so far.
CPUs are plugged at logical(thread) cpu level, one at a time.
And ACPI spec does the same (describes logical CPUs) and hotplug
notification in guest handled per one logical cpu at a time.

> objected to doing it on socket level. Bharata(?) had a recent patch
> to catch such incompletely filled cores on the initial command line
> and I think we should seriously consider doing that even if it breaks
> some fringe use case - hot-added incomplete cores or sockets remain
> to be detected.
interesting question, what MADT/SRAT tables would look like on machine
with Intel CPU when hyperthreading is disabled in BIOS? 
 
> 
> Regards,
> Andreas
> 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Wed, 16 Dec 2015 18:22:20 +0100
Igor Mammedov  wrote:

> On Wed, 16 Dec 2015 16:57:54 +0100
> Andreas Färber  wrote:
[...]
> > 
> > Attendees in Seattle said that thread-level hot-plug were dangerous
> > for Linux guests due to assumptions in the (guest's) scheduler
> > breaking for any incompletely filled cores or sockets. No one
> > present
> There is not such thing as cpu hotplug at socket level in x86 linux
> so far. CPUs are plugged at logical(thread) cpu level, one at a time.
> And ACPI spec does the same (describes logical CPUs) and hotplug
> notification in guest handled per one logical cpu at a time.
BTW:
from my experience with CPU hotplug, I've had to fix only one
topology+sched related bug in kernel but more bugs were related to fast
CPU online/offline especially if host is over-committed.

And plugging whole socket would lead to a rapid sequence of hotplugging
CPUs which has been causing non deterministic kernel crash/lock up
much more often than topology issues, there are still bugs to fix in
that area.

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Wed, 16 Dec 2015 16:46:37 +0100
Andreas Färber  wrote:

> Am 10.12.2015 um 13:35 schrieb Igor Mammedov:
> > wrt CLI can't we do something like this?
> > 
> > -device some-cpu-model,socket=x[,core=y[,thread=z]]
> 
> That's problematic and where my x86 remodeling got stuck. It works
> fine (more or less) to model sockets, cores and hyperthreads for
> -smp, but doing it dynamically did not work well. How do you
> determine the instance size a socket with N cores and M threads
> needs?
-smp defines necessary topology, all one need is to find out
instance size of a core or thread object, it would be probably x86
specific but it's doable, the only thing in current X86 thread
needs to fix is to reserve space for largest APIC type
and replace object_new with object_initialize_with_type.

However if I look from x86 point of view, there isn't need to model
sockets nor cores. The threads are sufficient for QEMU needs.
Dummy sockets/cores are just complicating implementation.

What I'm advocating for is let archs to decide if they should create
CPUs per socket, core or thread.

And for x86 do this at thread level, that way we keep compatibility
with cpu-add, but will also allow which cpu thread to plug with
 'node=n,socket=x,core=y,thread=z'.

Another point in favor of thread granularity for x86 is that competing
hyper-visors are doing that at thread level, QEMU would be worse off
in feature parity if minimal hotplug unit would be a socket. 

That also has benefit of being very flexible and would also suit
engineering audience of QEMU, allowing them to build CPUs from
config instead of hardcoding it in code and playing heterogeneous
configurations.

Options 'node=n,socket=x,core=y,thread=z' are just a SMP specific
path, defining where CPU should attached, it could be a QOM path
in the future when we arrive there and have a stable QOM tree. 

> Allocations in instance_init are to be avoided with a view to
> hot-plug.
> So either we have a fully determined socket object or we
> need to wire individual objects on the command line. The latter has
> bad implications for atomicity and thus hot-unplug. That leaves us
what are these bad implication and how they affect unplug?

If for example x86 CPU thread is fixed to embed child APIC then
to avoid allocations as much as possible or fail gracefully there
is 2 options :
 1: like you've said, reserve all needed space at startup, i.e.
pre-create empty sockets
 2: fail gracefully in qdev_device_add() if allocation is not possible

for #2 it's not enough to avoid allocations in instance_init()
we also must teach qdev_device_add() to get the size of to be created
object and replace
object_new() with malloc() + object_initialize_with_type(),
that way it's possible to fail allocation gracefully and report error.

Doing that would benefit not only CPUs but every device_add capable
Device and is sufficient for hotplug purposes without overhead of
reserving space for every possible hotplugged device at startup (which
is impossible anyway in generic)
So I'd go for #2 sane device_add impl. vs #1 preallocated objects one  
 
> with dynamic properties doing allocations and reporting it via
> Error**, something I never finished and could use reviewers and
> contributors.
most of dynamic properties are static, looks like what QOM needs
is really static properties so we don't misuse the former and probably
a way to reserve space for declared number of dynamic ones to avoid
allocations in instance_initialize().

> 
> Anthony's old suggestion had been to use real socket product names
> like Xeon-E5-4242 to get a 6-core, dual-thread socket, without
> parameters - unfortunately I still don't see an easy way to define
> such a thing today with the flexibility users will undoubtedly want.
I don't see it either and for me it is much harder to remember
what Xeon-E5-4242 is while it's much easier to say:
   I want N [cpu-foo] threads
which in SMP world could be expressed via add N tread objects at
specified locations
   device_add cpu-foo, with optional node=n,socket=x,core=y,thread=z
allows to do it.
And well for x86 there lots of these Xeon-foo/whatever-foo codenames,
which would be nightmare to maintain.

> 
> And since the question came up how to detect this, what you guys seem
> to keep forgetting is that somewhere there also needs to be a matching
> link<> property that determines what can be plugged, i.e. QMP
> qom-list. link<>s are the QOM equivalent to qdev's buses. The object
> itself needs to live in /machine/peripheral
> or /machine/peripheral-anon (/machine/unattached is supposed to go
> away after the QOM conversion is done!) and in a machine-specific
> place there will be a /machine/cpu-socket[0]
> -> /machine/peripheral-anon/device[42] link
> property. It might just as well
> be /machine/daughterboard-x/cpu-core[2] -> /machine/peripheral/cpu0.
> (Gentle reminder of the s390 ipi modeling discussion that never came
> to any conclusion iirc.)
QOM view probably is 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Fri, 11 Dec 2015 09:27:57 +0530
Bharata B Rao  wrote:

> On Thu, Dec 10, 2015 at 01:35:05PM +0100, Igor Mammedov wrote:
> > On Thu, 10 Dec 2015 11:45:35 +0530
> > Bharata B Rao  wrote:
> > 
> > > Hi,
> > > 
> > > This is an attempt to define a generic CPU device that serves as a
> > > containing device to underlying arch-specific CPU devices. The
> > > motivation for this is to have an arch-neutral way to specify CPUs
> > > mainly during hotplug.
> > > 
> > > Instead of individual archs having their own semantics to specify
> > > the CPU like
> > > 
> > > -device POWER8-powerpc64-cpu (pseries)
> > > -device qemu64-x86_64-cpu (pc)
> > > -device s390-cpu (s390)
> > > 
> > > this patch introduces a new device named cpu-core that could be
> > > used for all target archs as
> > > 
> > > -device cpu-core,socket="sid"
> > > 
> > > This adds a CPU core with all its associated threads into the
> > > specified socket with id "sid". The number of target architecture
> > > specific CPU threads that get created during this operation is
> > > based on the CPU topology specified using -smp
> > > sockets=S,cores=C,threads=T option. Also the number of cores that
> > > can be accommodated in the same socket is dictated by the cores=
> > > parameter in the same -smp option.
> > > 
> > > CPU sockets are represented by QOM objects and the number of
> > > sockets required to fit in max_cpus are created at boottime. As
> > > cpu-core devices are created, they are linked to socket object
> > > specified by socket="sid" device property.
> > > 
> > > Thus the model consists of backend socket objects which can be
> > > considered as container of one or more cpu-core devices. Each
> > > cpu-core object is linked to the appropriate backend socket
> > > object. Each CPU thread device appears as child object of
> > > cpu-core device.
> > > 
> > > All the required socket objects are created upfront and they
> > > can't be deleted. Though currently socket objects can be created
> > > using object_add monitor command, I am planning to prevent that
> > > so that a guest boots with the required number of sockets and
> > > only CPU cores can be hotplugged into them.
> > > 
> > > CPU hotplug granularity
> > > ---
> > > CPU hotplug will now be done in cpu-core device granularity.
> > > 
> > > This patchset includes a patch to prevent topologies that result
> > > in partially filled cores. Hence with this patchset, we will
> > > always have fully filled cpu-core devices both for boot time and
> > > during hotplug.
> > > 
> > > For archs like PowerPC, where there is no requirement to be fully
> > > similar to the physical system, hotplugging CPU at core
> > > granularity is common. While core level hotplug will fit in
> > > naturally for such archs, for others which want socket level
> > > hotplug, could higher level tools like libvirt perform multiple
> > > core hotplugs in response to one socket hotplug request ?
> > > 
> > > Are there archs that would need thread level CPU addition ?
> > there are,
> > currently x86 target allows to start QEMU with 1 thread even if
> > topology specifies more threads per core. The same applies to
> > hotplug.
> > 
> > On top of that I think ACPI spec also treats CPU devices on per
> > threads level.
> 
> > with this series it would regress following CLI
> >   -smp 1,sockets=4,cores=2,threads=2,maxcpus=16
> 
> Yes, the first patch in this series explicitly prevents such
> topologies.
> 
> Though QEMU currently allows to have such topologies, as discussed
> at http://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00396.html
> is there a need to continue support for such topologies ?
>  
> > 
> > wrt CLI can't we do something like this?
> > 
> > -device some-cpu-model,socket=x[,core=y[,thread=z]]
> 
> We can, I just started with a simple homogenous setup. As David Gibson
> pointed out elsewhere, instead of taking the topology from globals,
> making it part of each -device command line like you show above would
> pave way for heterogenous setup which probably would be needed in
> future. In such a case, we wouldn't anyway have to debate about
> supporting topologies with partially filled cores and sockets. Also
> supporting legacy cpu-add x86 via device_add method would probably be
> easier with the semantics you showed.
> 
> > 
> > for NUMA configs individual sockets IDs could be bound to
> > nodes via -numa ... option
> 
> For PowerPC, socket is not always a NUMA boundary, it can have
> two CPU packages/chips (DCM)  within a socket and hence have two NUMA
> levels within a socket.
I guess syntax could be extended with an optional 'numa' option,
like we do with pc-dimm device:

-device some-cpu-model,[numa=n,]socket=x[,core=y[,thread=z]]

s/some-cpu-model/some-cpu-type/

but it should be ok if libvirt would feed 'numa' option when creating
cpu device.

> 
> > 
> > and allow individual targets to use its own way to build CPUs?
> > 
> 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Andreas Färber]

Am 15.12.2015 um 06:27 schrieb Zhu Guihua:
> 
>>> and allow individual targets to use its own way to build CPUs?
>>>
>>> For initial conversion of x86-cpus to device-add we could do pretty
>>> much the same like we do now, where cpu devices will appear under:
>>> /machine (pc-i440fx-2.5-machine)
>>>/unattached (container)
>>>  /device[x] (qemu64-x86_64-cpu)
>>>
>>> since we don't have to maintain/model dummy socket/core objects.
>>>
>>> PowerPC could do the similar only at core level since it has
>>> need for modeling core objects.
>>>
>>> It doesn't change anything wrt current introspection state, since
>>> cpus could be still found by mgmt tools that parse QOM tree.
>>>
>>> We probably should split 2 conflicting goals we are trying to meet here,
>>>
>>>   1. make device-add/dell work with cpus /
>>>   drop support for cpu-add in favor of device_add
>>>
>>>   2. how to model QOM tree view for CPUs in arch independent manner
>>>  to make mgmt layer life easier.
>>>
>>> and work on them independently instead of arguing for years,
>>> that would allow us to make progress in #1 while still thinking about
>>> how to do #2 the right way if we really need it.
>> Makes sense, s390 developer also recommends the same. Given that we have
>> CPU hotplug patchsets from x86, PowerPC and s390 all implementing
>> device_add
>> semantics pending on the list, can we hope to get them merged for
>> QEMU-2.6 ?
>>
>> So as seen below, the device is either "cpu_model-cpu_type" or just
>> "cpu_type".
>>
>> -device POWER8-powerpc64-cpu (pseries)
>> -device qemu64-x86_64-cpu (pc)
>> -device s390-cpu (s390)
>>
>> Is this going to be the final acceptable semantics ? Would libvirt be
>> able
>> to work with this different CPU device names for different guests ?
> 
> Is operating on core level not final decision ?

No, it is absolutely _not_ the conclusion from Seattle.

Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg)

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Wed, 16 Dec 2015 16:19:06 +0100
Andreas Färber  wrote:

> Am 10.12.2015 um 07:15 schrieb Bharata B Rao:
> > CPU hotplug granularity
> > ---
> > CPU hotplug will now be done in cpu-core device granularity.
> 
> Nack.
> 
> > Are there archs that would need thread level CPU addition ?
> 
> Yes, s390. And for x86 people called for socket level.
socket level hotplug would be the last resort if we can't agree
on thread level one. As it would break existing setups where
user can hotplug 1 core, and  I'd like to avoid it if it is possible.

> 
> Andreas
> 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Andreas Färber]

Am 10.12.2015 um 07:15 schrieb Bharata B Rao:
> CPU hotplug granularity
> ---
> CPU hotplug will now be done in cpu-core device granularity.

Nack.

> Are there archs that would need thread level CPU addition ?

Yes, s390. And for x86 people called for socket level.

Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg)

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Andreas Färber]

Am 10.12.2015 um 13:35 schrieb Igor Mammedov:
> wrt CLI can't we do something like this?
> 
> -device some-cpu-model,socket=x[,core=y[,thread=z]]

That's problematic and where my x86 remodeling got stuck. It works fine
(more or less) to model sockets, cores and hyperthreads for -smp, but
doing it dynamically did not work well. How do you determine the
instance size a socket with N cores and M threads needs? Allocations in
instance_init are to be avoided with a view to hot-plug. So either we
have a fully determined socket object or we need to wire individual
objects on the command line. The latter has bad implications for
atomicity and thus hot-unplug. That leaves us with dynamic properties
doing allocations and reporting it via Error**, something I never
finished and could use reviewers and contributors.

Anthony's old suggestion had been to use real socket product names like
Xeon-E5-4242 to get a 6-core, dual-thread socket, without parameters -
unfortunately I still don't see an easy way to define such a thing today
with the flexibility users will undoubtedly want.

And since the question came up how to detect this, what you guys seem to
keep forgetting is that somewhere there also needs to be a matching
link<> property that determines what can be plugged, i.e. QMP qom-list.
link<>s are the QOM equivalent to qdev's buses. The object itself needs
to live in /machine/peripheral or /machine/peripheral-anon
(/machine/unattached is supposed to go away after the QOM conversion is
done!) and in a machine-specific place there will be a
/machine/cpu-socket[0] -> /machine/peripheral-anon/device[42]
link property. It might just as well be
/machine/daughterboard-x/cpu-core[2] -> /machine/peripheral/cpu0.
(Gentle reminder of the s390 ipi modeling discussion that never came to
any conclusion iirc.)

Note that I have not read this patch series yet, just some of the
alarming review comments.

Regards,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg)

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Andreas Färber]

Am 16.12.2015 um 16:44 schrieb Igor Mammedov:
> On Wed, 16 Dec 2015 16:19:06 +0100
> Andreas Färber  wrote:
> 
>> Am 10.12.2015 um 07:15 schrieb Bharata B Rao:
>>> CPU hotplug granularity
>>> ---
>>> CPU hotplug will now be done in cpu-core device granularity.
>>
>> Nack.
>>
>>> Are there archs that would need thread level CPU addition ?
>>
>> Yes, s390. And for x86 people called for socket level.
> socket level hotplug would be the last resort if we can't agree
> on thread level one. As it would break existing setups where
> user can hotplug 1 core, and  I'd like to avoid it if it is possible.

We still need to keep cpu-add for backwards compatibility, so I am
discussing solely the new device_add interface. My previous x86 series
went to severe hacks trying to keep cpu-add working with sockets

Attendees in Seattle said that thread-level hot-plug were dangerous for
Linux guests due to assumptions in the (guest's) scheduler breaking for
any incompletely filled cores or sockets. No one present objected to
doing it on socket level. Bharata(?) had a recent patch to catch such
incompletely filled cores on the initial command line and I think we
should seriously consider doing that even if it breaks some fringe use
case - hot-added incomplete cores or sockets remain to be detected.

Regards,
Andreas

-- 
SUSE Linux GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Felix Imendörffer, Jane Smithard, Graham Norton; HRB 21284 (AG Nürnberg)

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Zhu Guihua]

and allow individual targets to use its own way to build CPUs?

For initial conversion of x86-cpus to device-add we could do pretty
much the same like we do now, where cpu devices will appear under:
/machine (pc-i440fx-2.5-machine)
   /unattached (container)
 /device[x] (qemu64-x86_64-cpu)

since we don't have to maintain/model dummy socket/core objects.

PowerPC could do the similar only at core level since it has
need for modeling core objects.

It doesn't change anything wrt current introspection state, since
cpus could be still found by mgmt tools that parse QOM tree.

We probably should split 2 conflicting goals we are trying to meet here,

  1. make device-add/dell work with cpus /
  drop support for cpu-add in favor of device_add

  2. how to model QOM tree view for CPUs in arch independent manner
 to make mgmt layer life easier.

and work on them independently instead of arguing for years,
that would allow us to make progress in #1 while still thinking about
how to do #2 the right way if we really need it.

Makes sense, s390 developer also recommends the same. Given that we have
CPU hotplug patchsets from x86, PowerPC and s390 all implementing device_add
semantics pending on the list, can we hope to get them merged for
QEMU-2.6 ?

So as seen below, the device is either "cpu_model-cpu_type" or just "cpu_type".

-device POWER8-powerpc64-cpu (pseries)
-device qemu64-x86_64-cpu (pc)
-device s390-cpu (s390)

Is this going to be the final acceptable semantics ? Would libvirt be able
to work with this different CPU device names for different guests ?


Is operating on core level not final decision ?

For progress, I also agree to implement device_add for different archs.

Thanks,
Zhu

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Bharata B Rao]

On Thu, Dec 10, 2015 at 03:25:53PM -0500, Matthew Rosato wrote:
> On 12/10/2015 01:15 AM, Bharata B Rao wrote:
> > Hi,
> > 
> > This is an attempt to define a generic CPU device that serves as a
> > containing device to underlying arch-specific CPU devices. The motivation
> > for this is to have an arch-neutral way to specify CPUs mainly during
> > hotplug.
> > 
> > Instead of individual archs having their own semantics to specify the
> > CPU like
> > 
> > -device POWER8-powerpc64-cpu (pseries)
> > -device qemu64-x86_64-cpu (pc)
> > -device s390-cpu (s390)
> > 
> > this patch introduces a new device named cpu-core that could be
> > used for all target archs as
> > 
> > -device cpu-core,socket="sid"
> > 
> > This adds a CPU core with all its associated threads into the specified
> > socket with id "sid". The number of target architecture specific CPU threads
> > that get created during this operation is based on the CPU topology 
> > specified
> > using -smp sockets=S,cores=C,threads=T option. Also the number of cores that
> > can be accommodated in the same socket is dictated by the cores= parameter
> > in the same -smp option.
> > 
> > CPU sockets are represented by QOM objects and the number of sockets 
> > required
> > to fit in max_cpus are created at boottime. As cpu-core devices are
> > created, they are linked to socket object specified by socket="sid" device
> > property.
> > 
> > Thus the model consists of backend socket objects which can be considered
> > as container of one or more cpu-core devices. Each cpu-core object is
> > linked to the appropriate backend socket object. Each CPU thread device
> > appears as child object of cpu-core device.
> > 
> > All the required socket objects are created upfront and they can't be 
> > deleted.
> > Though currently socket objects can be created using object_add monitor
> > command, I am planning to prevent that so that a guest boots with the
> > required number of sockets and only CPU cores can be hotplugged into
> > them.
> > 
> > CPU hotplug granularity
> > ---
> > CPU hotplug will now be done in cpu-core device granularity.
> > 
> > This patchset includes a patch to prevent topologies that result in
> > partially filled cores. Hence with this patchset, we will always
> > have fully filled cpu-core devices both for boot time and during hotplug.
> > 
> > For archs like PowerPC, where there is no requirement to be fully
> > similar to the physical system, hotplugging CPU at core granularity
> > is common. While core level hotplug will fit in naturally for such
> > archs, for others which want socket level hotplug, could higher level
> > tools like libvirt perform multiple core hotplugs in response to one
> > socket hotplug request ?
> > 
> > Are there archs that would need thread level CPU addition ?
> > 
> > Boot time CPUs as cpu-core devices
> > --
> > In this patchset, I am coverting the boot time CPU initialization
> > (from -smp option) to initialize the required number of cpu-core
> > devices and linking them with the appropriate socket objects.
> > 
> > Initially I thought we should be able to completely replace -smp with
> > -device cpu-core, but then I realized that at least both x86 and pseries
> > guests' machine init code has dependencies on first CPU being available
> > for the machine init code to work correctly.
> > 
> > Currently I have converted boot CPUs to cpu-core devices only PowerPC sPAPR
> > and i386 PC targets. I am not really sure about the i386 changes and the
> > intention in this iteration was to check if it is indeed possible to
> > fit i386 into cpu-core model. Having said that I am able to boot an x86
> > guest with this patchset.
> 
> I attempted a quick conversion for s390 to using cpu-core, but looks
> like we'd have an issue preventing s390 from using cpu-core immediately
> -- it relies on cpu_generic_init, which s390 specifically avoids today
> because we don't have support for cpu_models.  Not sure if other
> architectures will have the same issue.

I see that there are a few archs that don't have cpu_model, I guess
we can teach cpu_generic_init() to handle that. I will attempt that
in the next version if needed.

Regards,
Bharata.

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Matthew Rosato]

On 12/10/2015 01:15 AM, Bharata B Rao wrote:
> Hi,
> 
> This is an attempt to define a generic CPU device that serves as a
> containing device to underlying arch-specific CPU devices. The motivation
> for this is to have an arch-neutral way to specify CPUs mainly during
> hotplug.
> 
> Instead of individual archs having their own semantics to specify the
> CPU like
> 
> -device POWER8-powerpc64-cpu (pseries)
> -device qemu64-x86_64-cpu (pc)
> -device s390-cpu (s390)
> 
> this patch introduces a new device named cpu-core that could be
> used for all target archs as
> 
> -device cpu-core,socket="sid"
> 
> This adds a CPU core with all its associated threads into the specified
> socket with id "sid". The number of target architecture specific CPU threads
> that get created during this operation is based on the CPU topology specified
> using -smp sockets=S,cores=C,threads=T option. Also the number of cores that
> can be accommodated in the same socket is dictated by the cores= parameter
> in the same -smp option.
> 
> CPU sockets are represented by QOM objects and the number of sockets required
> to fit in max_cpus are created at boottime. As cpu-core devices are
> created, they are linked to socket object specified by socket="sid" device
> property.
> 
> Thus the model consists of backend socket objects which can be considered
> as container of one or more cpu-core devices. Each cpu-core object is
> linked to the appropriate backend socket object. Each CPU thread device
> appears as child object of cpu-core device.
> 
> All the required socket objects are created upfront and they can't be deleted.
> Though currently socket objects can be created using object_add monitor
> command, I am planning to prevent that so that a guest boots with the
> required number of sockets and only CPU cores can be hotplugged into
> them.
> 
> CPU hotplug granularity
> ---
> CPU hotplug will now be done in cpu-core device granularity.
> 
> This patchset includes a patch to prevent topologies that result in
> partially filled cores. Hence with this patchset, we will always
> have fully filled cpu-core devices both for boot time and during hotplug.
> 
> For archs like PowerPC, where there is no requirement to be fully
> similar to the physical system, hotplugging CPU at core granularity
> is common. While core level hotplug will fit in naturally for such
> archs, for others which want socket level hotplug, could higher level
> tools like libvirt perform multiple core hotplugs in response to one
> socket hotplug request ?
> 
> Are there archs that would need thread level CPU addition ?
> 
> Boot time CPUs as cpu-core devices
> --
> In this patchset, I am coverting the boot time CPU initialization
> (from -smp option) to initialize the required number of cpu-core
> devices and linking them with the appropriate socket objects.
> 
> Initially I thought we should be able to completely replace -smp with
> -device cpu-core, but then I realized that at least both x86 and pseries
> guests' machine init code has dependencies on first CPU being available
> for the machine init code to work correctly.
> 
> Currently I have converted boot CPUs to cpu-core devices only PowerPC sPAPR
> and i386 PC targets. I am not really sure about the i386 changes and the
> intention in this iteration was to check if it is indeed possible to
> fit i386 into cpu-core model. Having said that I am able to boot an x86
> guest with this patchset.

I attempted a quick conversion for s390 to using cpu-core, but looks
like we'd have an issue preventing s390 from using cpu-core immediately
-- it relies on cpu_generic_init, which s390 specifically avoids today
because we don't have support for cpu_models.  Not sure if other
architectures will have the same issue.

I agree with Igor's sentiment of separating the issue of device_add
hotplug vs generic QOM view -- s390 could support device_add/del for
s390-cpu now, but the addition of cpu-core just adds more requirements
before we can allow for hotplug, without providing any immediate benefit
since s390 doesn't currently surface any topology info to the guest.

Matt

> 
> NUMA
> 
> TODO: In this patchset, I haven't explicitly done anything for NUMA yet.
> I am thinking if we could add node=N option to cpu-core device.
> That could specify the NUMA node to which the CPU core belongs to.
> 
> -device cpu-core,socket="sid",node=N
> 
> QOM composition tree
> -
> QOM composition tree for x86 where I don't have CPU hotplug enabled, but
> just initializing boot CPUs as cpu-core devices appears like this:
> 
> -smp 4,sockets=4,cores=2,threads=2,maxcpus=16
> 
> /machine (pc-i440fx-2.5-machine)
>   /unattached (container)
> /device[0] (cpu-core)
>   /thread[0] (qemu64-x86_64-cpu)
>   /thread[1] (qemu64-x86_64-cpu)
> /device[4] (cpu-core)
>   /thread[0] (qemu64-x86_64-cpu)
>   /thread[1] (qemu64-x86_64-cpu)
> 
> For PowerPC where I 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Bharata B Rao]

On Thu, Dec 10, 2015 at 01:35:05PM +0100, Igor Mammedov wrote:
> On Thu, 10 Dec 2015 11:45:35 +0530
> Bharata B Rao  wrote:
> 
> > Hi,
> > 
> > This is an attempt to define a generic CPU device that serves as a
> > containing device to underlying arch-specific CPU devices. The
> > motivation for this is to have an arch-neutral way to specify CPUs
> > mainly during hotplug.
> > 
> > Instead of individual archs having their own semantics to specify the
> > CPU like
> > 
> > -device POWER8-powerpc64-cpu (pseries)
> > -device qemu64-x86_64-cpu (pc)
> > -device s390-cpu (s390)
> > 
> > this patch introduces a new device named cpu-core that could be
> > used for all target archs as
> > 
> > -device cpu-core,socket="sid"
> > 
> > This adds a CPU core with all its associated threads into the
> > specified socket with id "sid". The number of target architecture
> > specific CPU threads that get created during this operation is based
> > on the CPU topology specified using -smp sockets=S,cores=C,threads=T
> > option. Also the number of cores that can be accommodated in the same
> > socket is dictated by the cores= parameter in the same -smp option.
> > 
> > CPU sockets are represented by QOM objects and the number of sockets
> > required to fit in max_cpus are created at boottime. As cpu-core
> > devices are created, they are linked to socket object specified by
> > socket="sid" device property.
> > 
> > Thus the model consists of backend socket objects which can be
> > considered as container of one or more cpu-core devices. Each
> > cpu-core object is linked to the appropriate backend socket object.
> > Each CPU thread device appears as child object of cpu-core device.
> > 
> > All the required socket objects are created upfront and they can't be
> > deleted. Though currently socket objects can be created using
> > object_add monitor command, I am planning to prevent that so that a
> > guest boots with the required number of sockets and only CPU cores
> > can be hotplugged into them.
> > 
> > CPU hotplug granularity
> > ---
> > CPU hotplug will now be done in cpu-core device granularity.
> > 
> > This patchset includes a patch to prevent topologies that result in
> > partially filled cores. Hence with this patchset, we will always
> > have fully filled cpu-core devices both for boot time and during
> > hotplug.
> > 
> > For archs like PowerPC, where there is no requirement to be fully
> > similar to the physical system, hotplugging CPU at core granularity
> > is common. While core level hotplug will fit in naturally for such
> > archs, for others which want socket level hotplug, could higher level
> > tools like libvirt perform multiple core hotplugs in response to one
> > socket hotplug request ?
> > 
> > Are there archs that would need thread level CPU addition ?
> there are,
> currently x86 target allows to start QEMU with 1 thread even if
> topology specifies more threads per core. The same applies to hotplug.
> 
> On top of that I think ACPI spec also treats CPU devices on per threads
> level.

> with this series it would regress following CLI
>   -smp 1,sockets=4,cores=2,threads=2,maxcpus=16

Yes, the first patch in this series explicitly prevents such topologies.

Though QEMU currently allows to have such topologies, as discussed
at http://lists.gnu.org/archive/html/qemu-devel/2015-12/msg00396.html
is there a need to continue support for such topologies ?
 
> 
> wrt CLI can't we do something like this?
> 
> -device some-cpu-model,socket=x[,core=y[,thread=z]]

We can, I just started with a simple homogenous setup. As David Gibson
pointed out elsewhere, instead of taking the topology from globals,
making it part of each -device command line like you show above would pave way
for heterogenous setup which probably would be needed in future. In such a
case, we wouldn't anyway have to debate about supporting topologies
with partially filled cores and sockets. Also supporting legacy cpu-add
x86 via device_add method would probably be easier with the semantics
you showed.

> 
> for NUMA configs individual sockets IDs could be bound to
> nodes via -numa ... option

For PowerPC, socket is not always a NUMA boundary, it can have
two CPU packages/chips (DCM)  within a socket and hence have two NUMA
levels within a socket.

> 
> and allow individual targets to use its own way to build CPUs?
> 
> For initial conversion of x86-cpus to device-add we could do pretty
> much the same like we do now, where cpu devices will appear under:
> /machine (pc-i440fx-2.5-machine)
>   /unattached (container)
> /device[x] (qemu64-x86_64-cpu)
> 
> since we don't have to maintain/model dummy socket/core objects.
> 
> PowerPC could do the similar only at core level since it has
> need for modeling core objects.
> 
> It doesn't change anything wrt current introspection state, since
> cpus could be still found by mgmt tools that parse QOM tree.
> 
> We probably should split 2 conflicting 

Re: [Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Igor Mammedov]

On Thu, 10 Dec 2015 11:45:35 +0530
Bharata B Rao  wrote:

> Hi,
> 
> This is an attempt to define a generic CPU device that serves as a
> containing device to underlying arch-specific CPU devices. The
> motivation for this is to have an arch-neutral way to specify CPUs
> mainly during hotplug.
> 
> Instead of individual archs having their own semantics to specify the
> CPU like
> 
> -device POWER8-powerpc64-cpu (pseries)
> -device qemu64-x86_64-cpu (pc)
> -device s390-cpu (s390)
> 
> this patch introduces a new device named cpu-core that could be
> used for all target archs as
> 
> -device cpu-core,socket="sid"
> 
> This adds a CPU core with all its associated threads into the
> specified socket with id "sid". The number of target architecture
> specific CPU threads that get created during this operation is based
> on the CPU topology specified using -smp sockets=S,cores=C,threads=T
> option. Also the number of cores that can be accommodated in the same
> socket is dictated by the cores= parameter in the same -smp option.
> 
> CPU sockets are represented by QOM objects and the number of sockets
> required to fit in max_cpus are created at boottime. As cpu-core
> devices are created, they are linked to socket object specified by
> socket="sid" device property.
> 
> Thus the model consists of backend socket objects which can be
> considered as container of one or more cpu-core devices. Each
> cpu-core object is linked to the appropriate backend socket object.
> Each CPU thread device appears as child object of cpu-core device.
> 
> All the required socket objects are created upfront and they can't be
> deleted. Though currently socket objects can be created using
> object_add monitor command, I am planning to prevent that so that a
> guest boots with the required number of sockets and only CPU cores
> can be hotplugged into them.
> 
> CPU hotplug granularity
> ---
> CPU hotplug will now be done in cpu-core device granularity.
> 
> This patchset includes a patch to prevent topologies that result in
> partially filled cores. Hence with this patchset, we will always
> have fully filled cpu-core devices both for boot time and during
> hotplug.
> 
> For archs like PowerPC, where there is no requirement to be fully
> similar to the physical system, hotplugging CPU at core granularity
> is common. While core level hotplug will fit in naturally for such
> archs, for others which want socket level hotplug, could higher level
> tools like libvirt perform multiple core hotplugs in response to one
> socket hotplug request ?
> 
> Are there archs that would need thread level CPU addition ?
there are,
currently x86 target allows to start QEMU with 1 thread even if
topology specifies more threads per core. The same applies to hotplug.

On top of that I think ACPI spec also treats CPU devices on per threads
level.

> 
> Boot time CPUs as cpu-core devices
> --
> In this patchset, I am coverting the boot time CPU initialization
> (from -smp option) to initialize the required number of cpu-core
> devices and linking them with the appropriate socket objects.
> 
> Initially I thought we should be able to completely replace -smp with
> -device cpu-core, but then I realized that at least both x86 and
> pseries guests' machine init code has dependencies on first CPU being
> available for the machine init code to work correctly.
> 
> Currently I have converted boot CPUs to cpu-core devices only PowerPC
> sPAPR and i386 PC targets. I am not really sure about the i386
> changes and the intention in this iteration was to check if it is
> indeed possible to fit i386 into cpu-core model. Having said that I
> am able to boot an x86 guest with this patchset.
> 
> NUMA
> 
> TODO: In this patchset, I haven't explicitly done anything for NUMA
> yet. I am thinking if we could add node=N option to cpu-core device.
> That could specify the NUMA node to which the CPU core belongs to.
> 
> -device cpu-core,socket="sid",node=N
> 
> QOM composition tree
> -
> QOM composition tree for x86 where I don't have CPU hotplug enabled,
> but just initializing boot CPUs as cpu-core devices appears like this:
> 
> -smp 4,sockets=4,cores=2,threads=2,maxcpus=16
with this series it would regress following CLI
  -smp 1,sockets=4,cores=2,threads=2,maxcpus=16


wrt CLI can't we do something like this?

-device some-cpu-model,socket=x[,core=y[,thread=z]]

for NUMA configs individual sockets IDs could be bound to
nodes via -numa ... option

and allow individual targets to use its own way to build CPUs?

For initial conversion of x86-cpus to device-add we could do pretty
much the same like we do now, where cpu devices will appear under:
/machine (pc-i440fx-2.5-machine)
  /unattached (container)
/device[x] (qemu64-x86_64-cpu)

since we don't have to maintain/model dummy socket/core objects.

PowerPC could do the similar only at core level since it has
need for 

[Qemu-devel] [RFC PATCH v0 0/9] Generic cpu-core device

[Bharata B Rao]

Hi,

This is an attempt to define a generic CPU device that serves as a
containing device to underlying arch-specific CPU devices. The motivation
for this is to have an arch-neutral way to specify CPUs mainly during
hotplug.

Instead of individual archs having their own semantics to specify the
CPU like

-device POWER8-powerpc64-cpu (pseries)
-device qemu64-x86_64-cpu (pc)
-device s390-cpu (s390)

this patch introduces a new device named cpu-core that could be
used for all target archs as

-device cpu-core,socket="sid"

This adds a CPU core with all its associated threads into the specified
socket with id "sid". The number of target architecture specific CPU threads
that get created during this operation is based on the CPU topology specified
using -smp sockets=S,cores=C,threads=T option. Also the number of cores that
can be accommodated in the same socket is dictated by the cores= parameter
in the same -smp option.

CPU sockets are represented by QOM objects and the number of sockets required
to fit in max_cpus are created at boottime. As cpu-core devices are
created, they are linked to socket object specified by socket="sid" device
property.

Thus the model consists of backend socket objects which can be considered
as container of one or more cpu-core devices. Each cpu-core object is
linked to the appropriate backend socket object. Each CPU thread device
appears as child object of cpu-core device.

All the required socket objects are created upfront and they can't be deleted.
Though currently socket objects can be created using object_add monitor
command, I am planning to prevent that so that a guest boots with the
required number of sockets and only CPU cores can be hotplugged into
them.

CPU hotplug granularity
---
CPU hotplug will now be done in cpu-core device granularity.

This patchset includes a patch to prevent topologies that result in
partially filled cores. Hence with this patchset, we will always
have fully filled cpu-core devices both for boot time and during hotplug.

For archs like PowerPC, where there is no requirement to be fully
similar to the physical system, hotplugging CPU at core granularity
is common. While core level hotplug will fit in naturally for such
archs, for others which want socket level hotplug, could higher level
tools like libvirt perform multiple core hotplugs in response to one
socket hotplug request ?

Are there archs that would need thread level CPU addition ?

Boot time CPUs as cpu-core devices
--
In this patchset, I am coverting the boot time CPU initialization
(from -smp option) to initialize the required number of cpu-core
devices and linking them with the appropriate socket objects.

Initially I thought we should be able to completely replace -smp with
-device cpu-core, but then I realized that at least both x86 and pseries
guests' machine init code has dependencies on first CPU being available
for the machine init code to work correctly.

Currently I have converted boot CPUs to cpu-core devices only PowerPC sPAPR
and i386 PC targets. I am not really sure about the i386 changes and the
intention in this iteration was to check if it is indeed possible to
fit i386 into cpu-core model. Having said that I am able to boot an x86
guest with this patchset.

NUMA

TODO: In this patchset, I haven't explicitly done anything for NUMA yet.
I am thinking if we could add node=N option to cpu-core device.
That could specify the NUMA node to which the CPU core belongs to.

-device cpu-core,socket="sid",node=N

QOM composition tree
-
QOM composition tree for x86 where I don't have CPU hotplug enabled, but
just initializing boot CPUs as cpu-core devices appears like this:

-smp 4,sockets=4,cores=2,threads=2,maxcpus=16

/machine (pc-i440fx-2.5-machine)
  /unattached (container)
/device[0] (cpu-core)
  /thread[0] (qemu64-x86_64-cpu)
  /thread[1] (qemu64-x86_64-cpu)
/device[4] (cpu-core)
  /thread[0] (qemu64-x86_64-cpu)
  /thread[1] (qemu64-x86_64-cpu)

For PowerPC where I have CPU hotplug enabled:

-smp 4,sockets=4,cores=2,threads=2,maxcpus=16 -device 
cpu-core,socket=cpu-socket1,id=core3

/machine (pseries-2.5-machine)
  /unattached (container)
/device[1] (cpu-core)
  /thread[0] (host-powerpc64-cpu)
  /thread[1] (host-powerpc64-cpu)
/device[2] (cpu-core)
  /thread[0] (host-powerpc64-cpu)
  /thread[1] (host-powerpc64-cpu)
  /peripheral (container)
/core3 (cpu-core)
  /thread[0] (host-powerpc64-cpu)
  /thread[1] (host-powerpc64-cpu)

As can be seen, the boot CPU and hotplugged CPU come under separate
parents. Guess I should work towards getting both boot time and hotplugged
CPUs under same parent ?

Socket ID generation
-
In the current approach the socket ID generation is implicit somewhat.
All the sockets objects are created with pre-fixed format for ids like
cpu-socket0, cpu-socket1 etc. And machine init code of each arch