Re: [Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

On Wed, May 30, 2018 at 12:47:17PM +0200, Igor Mammedov wrote:
> Eduardo,
> 
> I've rebased series on top of current master
> the only change in several patches was s/2.13/3.0/
> otherwise there weren't any other conflicts.
> You can find rebased version at
> 
> https://github.com/imammedo/qemu.git qmp_preconfig_v9

Thanks!  I've rebased my branch on top of it and noticed that the
only differences are the "since 3.0" lines.  I've fixed them and
pushed a new numa-next branch.

Once the travis-ci job finishes, I will send a pull request.

-- 
Eduardo

Re: [Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

Eduardo,

I've rebased series on top of current master
the only change in several patches was s/2.13/3.0/
otherwise there weren't any other conflicts.
You can find rebased version at

https://github.com/imammedo/qemu.git qmp_preconfig_v9

Re: [Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

On Thu, 24 May 2018 17:42:03 +0200
Markus Armbruster  wrote:

> Eduardo Habkost  writes:
> 
> > I'm queueing this version (including v8 of patches 5-7) on
> > numa-next.
> >
> > Markus, Daniel: you were the people I remember expressing some
> > concerns about the new preconfig mechanism.  Do you have any
> > objections to this version?  
> 
> My gut's reaction to preconfig is as nauseous as ever.  It adds
> complexity we wouldn't want or need with sanely structured program
> startup.  It might still be the best we can do, and something we need to
> do.  You have to take on technical debt sometimes.  To know for sure
> this is the case here, we'd have to explore alternatives more seriously.
> Like repaying some of the existing technical debt around there.  Sadly,
> I lack the time to take on such a project.
Usually I do quite a bit of re-factoring to reduce our technical debt,
so while working on enabling VM start with libvirt using only --preconfig
(without -S) I most likely will have to improve code around VM creation.

[...]

Re: [Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Markus Armbruster]

Eduardo Habkost  writes:

> I'm queueing this version (including v8 of patches 5-7) on
> numa-next.
>
> Markus, Daniel: you were the people I remember expressing some
> concerns about the new preconfig mechanism.  Do you have any
> objections to this version?

My gut's reaction to preconfig is as nauseous as ever.  It adds
complexity we wouldn't want or need with sanely structured program
startup.  It might still be the best we can do, and something we need to
do.  You have to take on technical debt sometimes.  To know for sure
this is the case here, we'd have to explore alternatives more seriously.
Like repaying some of the existing technical debt around there.  Sadly,
I lack the time to take on such a project.

Since I lack the money to put where my mouth is, I'm going to shut up
and get out of your way.

Re: [Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Eduardo Habkost]

I'm queueing this version (including v8 of patches 5-7) on
numa-next.

Markus, Daniel: you were the people I remember expressing some
concerns about the new preconfig mechanism.  Do you have any
objections to this version?


On Fri, May 04, 2018 at 10:37:38AM +0200, Igor Mammedov wrote:
> 
> 
> v6->v7:
>   * fix up wording in documentation and left-overs of 'reusing' 'cont' 
> command in v5
>   * split out preconfig runstate introduction into separate patch to make
> series more bisection friendly and put allow-preconfig patch before
> --preconfig CLI patch
>   * merge qapi schema tests patches into one
>   * rename allowed_in_preconfig flag to a shorter allow_preconfig
>   * rename QCO_ALLOWED_IN_PRECONFIG into shorter QCO_ALLOW_PRECONFIG to match
> allow_preconfig flag
>   * reuse an-oob-command for allow_preconfig flag testing and rename it to
> test-flags-command
> v5->v6:
>   * add exit-preconfig QMP command instead of overloading meaning of 'cont' 
> command
>   * add doc text to qemu-tech.texi about -S and --preconfig
>   * add numa configuration example into commit message of 10/11
>   * limit set-numa-node QMP command to preconfig mode
> v4->v5:
>   * rebase on top of OOB changes that's now in master
>   * s/qobject_to_qdict(/qobject_to(QDict,/
>   * s/-preconfig/--preconfig/
>   * s/2.12/2.13/
>   * s/parse_NumaOptions/set_numa_options/
>   * drop if (err) guard around error_propagate()
>   * move QCO_ALLOWED_IN_PRECONFIG and do_qmp_dispatch() runstate check
> from the later patch 'qapi: introduce new cmd option  
> "allowed-in-preconfig"'
> to here for better bissectability
>   * add TODO comment to '{ RUN_STATE_PRECONFIG, RUN_STATE_INMIGRATE }' 
> transition
>   * check for incoming && preconfig outside of option parsing loop
>   * add 'use QMP instead" to error message, suggesting user the right 
> interface to use
>   * allow query-command-line-options in preconfig state
>   * make sure that allowed-in-preconfig could be set only to True
>   * move out QCO_ALLOWED_IN_PRECONFIG check in do_qmp_dispatch() to
> earlier 'cli: add --preconfig option' patch
>   * 2 extra test patches
>  'tests: let qapi-schema tests detect allowed-in-preconfig'
>  'tests: add allowed-in-preconfig-test for qapi-schema'
> v3->v4:
>   * replace 'runstates' list in  QMP command with a single
> boolean 'ption allowed-in-preconfig' like it's done with
> 'allow-oob'. Which allows to simplify intrusive QAPI
> changes quite a lot. (Eric Blake )
>   * Make sure HMP is disbled for real, v3 was just printing
> error mesage but allowing command to be executed
> ("Dr. David Alan Gilbert" )
>   * Improve "cli: add -preconfig option" commit message,
> explain a bit more on semantics of new state/option.
>   * ithe rest of minor fixups suggested at v3 review
> (Eric Blake )
> PS:
>havn't impl. test for new option in
>  tests/qapi-schema/qapi-schema-test.json yet,
>can do it on top if approach is acceptable.
> v1->v3:
>   * introduce PRECONFIG runstate with -preconfig option.
> it's cleaner to manage transitions and do checks
> than reusing existing PRELAUNCH state.
>   * extend QAPI schema commands with 'runstates' keyword,
> so that it would be possible to specify in command definition
> when it is valid to execute.
> (python changes a bit hackery, since I have little to
>  no idea how it should work)
>   * add preconfig QMP and set-numa-node tests
>   * make mutually exclusive -preconfig and -incoming options,
> for simplicity sake. Shouldn't be problem as target can
> be starter with pure CLI, since mapping on source is
> already known.
>   * Drop HMP part and leave only QMP in preconfig state.
> 
> 
> Currently it's problematic to configure NUMA mapping for CPUs using "-numa 
> cpu="
> option, without restarting QEMU, first is to query board specific CPU layout
> with query-hotpluggable-cpus QMP command (which is function of -M and -smp CLI
> options). Mgmt side isn't happy with restarting QEMU to query configuration
> parameters first and then run actual instance, so it keeps using old cpu_index
> based option '-numa cpus' instead of new socket/core/thread-id based one.
> 
> Introduce a new '--preconfig' CLI option which allows pausing QEMU before
> machine_init() is run (preconfig state). So it would be possible to query
> possible CPUs layout, configure NUMA mapping based on query result with
> set-numa-node QMP command and continue executing  without restarting QEMU.
> 
> Difference between new --preconfig pause point vs -S is that the later pauses
> QEMU after machine is constructed and ready to run guest code or in process of
> incoming migration (essentially machine is in some running state (with paused
> VCPUs) and any action on it is considered as hotplug). At this point it's hard
> to configure or reconfigure parameters that affect 

[Qemu-devel] [PATCH v7 00/11] enable numa configuration before machine_init() from QMP

[Igor Mammedov]

v6->v7:
  * fix up wording in documentation and left-overs of 'reusing' 'cont' command 
in v5
  * split out preconfig runstate introduction into separate patch to make
series more bisection friendly and put allow-preconfig patch before
--preconfig CLI patch
  * merge qapi schema tests patches into one
  * rename allowed_in_preconfig flag to a shorter allow_preconfig
  * rename QCO_ALLOWED_IN_PRECONFIG into shorter QCO_ALLOW_PRECONFIG to match
allow_preconfig flag
  * reuse an-oob-command for allow_preconfig flag testing and rename it to
test-flags-command
v5->v6:
  * add exit-preconfig QMP command instead of overloading meaning of 'cont' 
command
  * add doc text to qemu-tech.texi about -S and --preconfig
  * add numa configuration example into commit message of 10/11
  * limit set-numa-node QMP command to preconfig mode
v4->v5:
  * rebase on top of OOB changes that's now in master
  * s/qobject_to_qdict(/qobject_to(QDict,/
  * s/-preconfig/--preconfig/
  * s/2.12/2.13/
  * s/parse_NumaOptions/set_numa_options/
  * drop if (err) guard around error_propagate()
  * move QCO_ALLOWED_IN_PRECONFIG and do_qmp_dispatch() runstate check
from the later patch 'qapi: introduce new cmd option  
"allowed-in-preconfig"'
to here for better bissectability
  * add TODO comment to '{ RUN_STATE_PRECONFIG, RUN_STATE_INMIGRATE }' 
transition
  * check for incoming && preconfig outside of option parsing loop
  * add 'use QMP instead" to error message, suggesting user the right interface 
to use
  * allow query-command-line-options in preconfig state
  * make sure that allowed-in-preconfig could be set only to True
  * move out QCO_ALLOWED_IN_PRECONFIG check in do_qmp_dispatch() to
earlier 'cli: add --preconfig option' patch
  * 2 extra test patches
 'tests: let qapi-schema tests detect allowed-in-preconfig'
 'tests: add allowed-in-preconfig-test for qapi-schema'
v3->v4:
  * replace 'runstates' list in  QMP command with a single
boolean 'ption allowed-in-preconfig' like it's done with
'allow-oob'. Which allows to simplify intrusive QAPI
changes quite a lot. (Eric Blake )
  * Make sure HMP is disbled for real, v3 was just printing
error mesage but allowing command to be executed
("Dr. David Alan Gilbert" )
  * Improve "cli: add -preconfig option" commit message,
explain a bit more on semantics of new state/option.
  * ithe rest of minor fixups suggested at v3 review
(Eric Blake )
PS:
   havn't impl. test for new option in
 tests/qapi-schema/qapi-schema-test.json yet,
   can do it on top if approach is acceptable.
v1->v3:
  * introduce PRECONFIG runstate with -preconfig option.
it's cleaner to manage transitions and do checks
than reusing existing PRELAUNCH state.
  * extend QAPI schema commands with 'runstates' keyword,
so that it would be possible to specify in command definition
when it is valid to execute.
(python changes a bit hackery, since I have little to
 no idea how it should work)
  * add preconfig QMP and set-numa-node tests
  * make mutually exclusive -preconfig and -incoming options,
for simplicity sake. Shouldn't be problem as target can
be starter with pure CLI, since mapping on source is
already known.
  * Drop HMP part and leave only QMP in preconfig state.


Currently it's problematic to configure NUMA mapping for CPUs using "-numa cpu="
option, without restarting QEMU, first is to query board specific CPU layout
with query-hotpluggable-cpus QMP command (which is function of -M and -smp CLI
options). Mgmt side isn't happy with restarting QEMU to query configuration
parameters first and then run actual instance, so it keeps using old cpu_index
based option '-numa cpus' instead of new socket/core/thread-id based one.

Introduce a new '--preconfig' CLI option which allows pausing QEMU before
machine_init() is run (preconfig state). So it would be possible to query
possible CPUs layout, configure NUMA mapping based on query result with
set-numa-node QMP command and continue executing  without restarting QEMU.

Difference between new --preconfig pause point vs -S is that the later pauses
QEMU after machine is constructed and ready to run guest code or in process of
incoming migration (essentially machine is in some running state (with paused
VCPUs) and any action on it is considered as hotplug). At this point it's hard
to configure or reconfigure parameters that affect machine_init() and later
stages. While the new --preconfig option pauses QEMU instance before
machine_init() and would allow to configure parameters as if doing it from CLI
but in interactve manner using QMP interface, which would allow introspecting
and configuring QEMU instance without restarting it. 

Initially only limited set of commands (that are ready to work with non
initialized machine or without it) are allowed in preconfig state:

   #existing commands: