Re: [PATCH V9 00/46] Live Update
Good morning,
On 7/10/23 10:10, Steven Sistare wrote:
On 6/12/2023 10:59 AM, Michael Galaxy wrote:
Hi Steve,
On 6/7/23 12:37, Steven Sistare wrote:
On 6/7/2023 11:55 AM, Michael Galaxy wrote:
Another option could be to expose "-migrate-mode-disable" (instead of enable)
and just enable all 3 modes by default,
since we are already required to switch from "normal" mode to a CPR-specific
mode when it is time to do a live update,
if the intention is to preserve the capability to completely prevent a running
QEMU from using these modes
before the VM starts up.
- Michael
On 6/6/23 17:15, Michael Galaxy wrote:
Hi Steve,
In the current design you have, we have to specify both the command line parameter
"-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
Is it possible to opt-in to the CPR mode just once over the monitor instead of
having to specify it twice on the command line?
This would also match the live migration model: You do not need to necessarily "opt
in" to live migration mode through
a command line parameter, you simply request it when you need to. Can CPR
behave the same way?
This would also make switching over to a CPR-capable version of QEMU much
simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would no
longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and then
work on a libvirt patch later when
we're ready to do so.
Comments?
Hi Michael,
Requiring -migrate-enable-mode allows qemu to initialize objects
differently, if necessary, so that migration for a mode is not blocked.
See callers of migrate_mode_enabled. There is only one so far, in
ram_block_add. If the mode is cpr-exec, then it creates anonymous ram
blocks using memfd_create, else using MAP_ANON. In the V7 series, this
was controlled by a '-machine memfd-alloc=on' option.
migrate-enable-mode is more future proof for the user. If something new must
initialize differently to support cpr, then it adds a call to
migrate_mode_enabled,
and the command line remains the same. However, I could be persuaded to go
either way.
OK, so it is cpr-exec that needs this option (because of ram block allocation),
not really cpr-reboot.
Could the option then be made to only be required for cpr-exec and not
cpr-reboot, then,
since cpr-reboot doesn't require that consideration?
In a different forum Juan said this is a memory issue, so it should be expressed
as a memory related option. So, I will delete -migrate-enable-mode and revert
back
to -machine memfd-alloc, as defined in the V7 patch series.
Acknowledged. I'm going to try to get my reviewed-by's in soon. Sorry I
haven't done it sooner.
We've finished testing these patches on our systems and are moving forward.
A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
option. It needs to know which mode will be used, in order to check a
mode-specific blocker list.
Still, only-cpr-capable is also optional. If and only if one needs this option,
the mode could be
specified as part of the option itself, rather than requiring an extra command
line parameter, no?
Yes, I will make that change.
- Steve
Acknowledged.
Re: [PATCH V9 00/46] Live Update
On 6/12/2023 10:59 AM, Michael Galaxy wrote:
> Hi Steve,
> On 6/7/23 12:37, Steven Sistare wrote:
>> On 6/7/2023 11:55 AM, Michael Galaxy wrote:
>>> Another option could be to expose "-migrate-mode-disable" (instead of
>>> enable) and just enable all 3 modes by default,
>>> since we are already required to switch from "normal" mode to a
>>> CPR-specific mode when it is time to do a live update,
>>> if the intention is to preserve the capability to completely prevent a
>>> running QEMU from using these modes
>>> before the VM starts up.
>>>
>>> - Michael
>>>
>>> On 6/6/23 17:15, Michael Galaxy wrote:
Hi Steve,
In the current design you have, we have to specify both the command line
parameter "-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
Is it possible to opt-in to the CPR mode just once over the monitor
instead of having to specify it twice on the command line?
This would also match the live migration model: You do not need to
necessarily "opt in" to live migration mode through
a command line parameter, you simply request it when you need to. Can CPR
behave the same way?
This would also make switching over to a CPR-capable version of QEMU much
simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would no
longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and
then work on a libvirt patch later when
we're ready to do so.
Comments?
>> Hi Michael,
>> Requiring -migrate-enable-mode allows qemu to initialize objects
>> differently, if necessary, so that migration for a mode is not blocked.
>> See callers of migrate_mode_enabled. There is only one so far, in
>> ram_block_add. If the mode is cpr-exec, then it creates anonymous ram
>> blocks using memfd_create, else using MAP_ANON. In the V7 series, this
>> was controlled by a '-machine memfd-alloc=on' option.
>>
>> migrate-enable-mode is more future proof for the user. If something new must
>> initialize differently to support cpr, then it adds a call to
>> migrate_mode_enabled,
>> and the command line remains the same. However, I could be persuaded to go
>> either way.
>
> OK, so it is cpr-exec that needs this option (because of ram block
> allocation), not really cpr-reboot.
> Could the option then be made to only be required for cpr-exec and not
> cpr-reboot, then,
> since cpr-reboot doesn't require that consideration?
In a different forum Juan said this is a memory issue, so it should be expressed
as a memory related option. So, I will delete -migrate-enable-mode and revert
back
to -machine memfd-alloc, as defined in the V7 patch series.
>> A secondary reason for -migrate-enable-mode is to support the
>> only-cpr-capable
>> option. It needs to know which mode will be used, in order to check a
>> mode-specific blocker list.
>
> Still, only-cpr-capable is also optional. If and only if one needs this
> option, the mode could be
> specified as part of the option itself, rather than requiring an extra
> command line parameter, no?
Yes, I will make that change.
- Steve
> Further, in many clouds (including ours), our VM-management is generational
> with the
> development of the software versioning, so we *always* run tests and know
> whether or not a VM is CPR-capable.
>
> If it is not CPR-capable, we would never run the VM in the first place, which
> means we would never
> really use that option at all.
>
> I do see the appeal of the option, but could we simplify it, since it is
> opt-in?
Re: [PATCH V9 00/46] Live Update
Hi Steve,
On 6/7/23 12:37, Steven Sistare wrote:
On 6/7/2023 11:55 AM, Michael Galaxy wrote:
Another option could be to expose "-migrate-mode-disable" (instead of enable)
and just enable all 3 modes by default,
since we are already required to switch from "normal" mode to a CPR-specific
mode when it is time to do a live update,
if the intention is to preserve the capability to completely prevent a running
QEMU from using these modes
before the VM starts up.
- Michael
On 6/6/23 17:15, Michael Galaxy wrote:
Hi Steve,
In the current design you have, we have to specify both the command line parameter
"-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
Is it possible to opt-in to the CPR mode just once over the monitor instead of
having to specify it twice on the command line?
This would also match the live migration model: You do not need to necessarily "opt
in" to live migration mode through
a command line parameter, you simply request it when you need to. Can CPR
behave the same way?
This would also make switching over to a CPR-capable version of QEMU much
simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would no
longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and then
work on a libvirt patch later when
we're ready to do so.
Comments?
Hi Michael,
Requiring -migrate-enable-mode allows qemu to initialize objects
differently, if necessary, so that migration for a mode is not blocked.
See callers of migrate_mode_enabled. There is only one so far, in
ram_block_add. If the mode is cpr-exec, then it creates anonymous ram
blocks using memfd_create, else using MAP_ANON. In the V7 series, this
was controlled by a '-machine memfd-alloc=on' option.
migrate-enable-mode is more future proof for the user. If something new must
initialize differently to support cpr, then it adds a call to
migrate_mode_enabled,
and the command line remains the same. However, I could be persuaded to go
either way.
OK, so it is cpr-exec that needs this option (because of ram block
allocation), not really cpr-reboot.
Could the option then be made to only be required for cpr-exec and not
cpr-reboot, then,
since cpr-reboot doesn't require that consideration?
A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
option. It needs to know which mode will be used, in order to check a
mode-specific blocker list.
Still, only-cpr-capable is also optional. If and only if one needs this
option, the mode could be
specified as part of the option itself, rather than requiring an extra
command line parameter, no?
Further, in many clouds (including ours), our VM-management is
generational with the
development of the software versioning, so we *always* run tests and
know whether or not a VM is CPR-capable.
If it is not CPR-capable, we would never run the VM in the first place,
which means we would never
really use that option at all.
I do see the appeal of the option, but could we simplify it, since it is
opt-in?
- Michael
- Steve
Re: [PATCH V9 00/46] Live Update
On 6/7/2023 11:55 AM, Michael Galaxy wrote:
> Another option could be to expose "-migrate-mode-disable" (instead of enable)
> and just enable all 3 modes by default,
> since we are already required to switch from "normal" mode to a CPR-specific
> mode when it is time to do a live update,
> if the intention is to preserve the capability to completely prevent a
> running QEMU from using these modes
> before the VM starts up.
>
> - Michael
>
> On 6/6/23 17:15, Michael Galaxy wrote:
>> Hi Steve,
>>
>> In the current design you have, we have to specify both the command line
>> parameter "-migrate-mode-enable cpr-reboot"
>> *and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
>>
>> Is it possible to opt-in to the CPR mode just once over the monitor instead
>> of having to specify it twice on the command line?
>> This would also match the live migration model: You do not need to
>> necessarily "opt in" to live migration mode through
>> a command line parameter, you simply request it when you need to. Can CPR
>> behave the same way?
>>
>> This would also make switching over to a CPR-capable version of QEMU much
>> simpler and would even make it work for
>> existing libvirt-managed guests as their command line parameters would no
>> longer need to change. This would allow us to
>> simply power-off and power-on existing VMs to make them CPR-capable and then
>> work on a libvirt patch later when
>> we're ready to do so.
>>
>>
>> Comments?
Hi Michael,
Requiring -migrate-enable-mode allows qemu to initialize objects
differently, if necessary, so that migration for a mode is not blocked.
See callers of migrate_mode_enabled. There is only one so far, in
ram_block_add. If the mode is cpr-exec, then it creates anonymous ram
blocks using memfd_create, else using MAP_ANON. In the V7 series, this
was controlled by a '-machine memfd-alloc=on' option.
migrate-enable-mode is more future proof for the user. If something new must
initialize differently to support cpr, then it adds a call to
migrate_mode_enabled,
and the command line remains the same. However, I could be persuaded to go
either way.
A secondary reason for -migrate-enable-mode is to support the only-cpr-capable
option. It needs to know which mode will be used, in order to check a
mode-specific blocker list.
- Steve
Re: [PATCH V9 00/46] Live Update
Another option could be to expose "-migrate-mode-disable" (instead of
enable) and just enable all 3 modes by default,
since we are already required to switch from "normal" mode to a
CPR-specific mode when it is time to do a live update,
if the intention is to preserve the capability to completely prevent a
running QEMU from using these modes
before the VM starts up.
- Michael
On 6/6/23 17:15, Michael Galaxy wrote:
Hi Steve,
In the current design you have, we have to specify both the command
line parameter "-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
Is it possible to opt-in to the CPR mode just once over the monitor
instead of having to specify it twice on the command line?
This would also match the live migration model: You do not need to
necessarily "opt in" to live migration mode through
a command line parameter, you simply request it when you need to. Can
CPR behave the same way?
This would also make switching over to a CPR-capable version of QEMU
much simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would
no longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable
and then work on a libvirt patch later when
we're ready to do so.
Comments?
- Michael
On 12/7/22 09:48, Steven Sistare wrote:
This series desperately needs review in its intersection with live migration.
The code in other areas has been reviewed and revised multiple times -- thank
you!
David, Juan, can you spare some time to review this? I have done my best to
order
the patches logically (see the labelled groups in this email), and to provide
complete and clear cover letter and commit messages. Can I do anything to
facilitate,
like doing a code walk through via zoom?
And of course, I welcome anyone's feedback.
Here is the original posting.
https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$
- Steve
On 7/26/2022 12:09 PM, Steve Sistare wrote:
This version of the live update patch series integrates live update into the
live migration framework. The new interfaces are:
* mode (migration parameter)
* cpr-exec-args (migration parameter)
* file (migration URI)
* migrate-mode-enable (command-line argument)
* only-cpr-capable (command-line argument)
Provide the cpr-exec and cpr-reboot migration modes for live update. These
save and restore VM state, with minimal guest pause time, so that qemu may be
updated to a new version in between. The caller sets the mode parameter
before invoking the migrate or migrate-incoming commands.
In cpr-reboot mode, the migrate command saves state to a file, allowing
one to quit qemu, reboot to an updated kernel, start an updated version of
qemu, and resume via the migrate-incoming command. The caller must specify
a migration URI that writes to and reads from a file. Unlike normal mode,
the use of certain local storage options does not block the migration, but
the caller must not modify guest block devices between the quit and restart.
The guest RAM memory-backend must be shared, and the @x-ignore-shared
migration capability must be set, to avoid saving it to the file. Guest RAM
must be non-volatile across reboot, which can be achieved by backing it with
a dax device, or /dev/shm PKRAM as proposed in
https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$
but this is not enforced. The restarted qemu arguments must match those used
to initially start qemu, plus the -incoming option.
The reboot mode supports vfio devices if the caller first suspends the guest,
such as by issuing guest-suspend-ram to the qemu guest agent. The guest
drivers' suspend methods flush outstanding requests and re-initialize the
devices, and thus there is no device state to save and restore. After
issuing migrate-incoming, the caller must issue a system_wakeup command to
resume.
In cpr-exec mode, the migrate command saves state to a file and directly
exec's a new version of qemu on the same host, replacing the original process
while retaining its PID. The caller must specify a migration URI that writes
to and reads from a file, and resumes execution via the migrate-incoming
command. Arguments for the new qemu process are taken from the cpr-exec-args
migration parameter, and must include the -incoming option.
Guest RAM must be backed by a memory backend with share=on, but cannot be
memory-backend-ram. The memory is re-mmap'd in the updated process, so guest
ram is efficiently preserved in place, albeit with new virtual addresses.
In addition, the '-migrate-mode-enable
Re: [PATCH V9 00/46] Live Update
Hi Steve,
In the current design you have, we have to specify both the command line
parameter "-migrate-mode-enable cpr-reboot"
*and* issue the monitor command "migrate_set_parameter mode cpr-${mode}".
Is it possible to opt-in to the CPR mode just once over the monitor
instead of having to specify it twice on the command line?
This would also match the live migration model: You do not need to
necessarily "opt in" to live migration mode through
a command line parameter, you simply request it when you need to. Can
CPR behave the same way?
This would also make switching over to a CPR-capable version of QEMU
much simpler and would even make it work for
existing libvirt-managed guests as their command line parameters would
no longer need to change. This would allow us to
simply power-off and power-on existing VMs to make them CPR-capable and
then work on a libvirt patch later when
we're ready to do so.
Comments?
- Michael
On 12/7/22 09:48, Steven Sistare wrote:
This series desperately needs review in its intersection with live migration.
The code in other areas has been reviewed and revised multiple times -- thank
you!
David, Juan, can you spare some time to review this? I have done my best to
order
the patches logically (see the labelled groups in this email), and to provide
complete and clear cover letter and commit messages. Can I do anything to
facilitate,
like doing a code walk through via zoom?
And of course, I welcome anyone's feedback.
Here is the original posting.
https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$
- Steve
On 7/26/2022 12:09 PM, Steve Sistare wrote:
This version of the live update patch series integrates live update into the
live migration framework. The new interfaces are:
* mode (migration parameter)
* cpr-exec-args (migration parameter)
* file (migration URI)
* migrate-mode-enable (command-line argument)
* only-cpr-capable (command-line argument)
Provide the cpr-exec and cpr-reboot migration modes for live update. These
save and restore VM state, with minimal guest pause time, so that qemu may be
updated to a new version in between. The caller sets the mode parameter
before invoking the migrate or migrate-incoming commands.
In cpr-reboot mode, the migrate command saves state to a file, allowing
one to quit qemu, reboot to an updated kernel, start an updated version of
qemu, and resume via the migrate-incoming command. The caller must specify
a migration URI that writes to and reads from a file. Unlike normal mode,
the use of certain local storage options does not block the migration, but
the caller must not modify guest block devices between the quit and restart.
The guest RAM memory-backend must be shared, and the @x-ignore-shared
migration capability must be set, to avoid saving it to the file. Guest RAM
must be non-volatile across reboot, which can be achieved by backing it with
a dax device, or /dev/shm PKRAM as proposed in
https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$
but this is not enforced. The restarted qemu arguments must match those used
to initially start qemu, plus the -incoming option.
The reboot mode supports vfio devices if the caller first suspends the guest,
such as by issuing guest-suspend-ram to the qemu guest agent. The guest
drivers' suspend methods flush outstanding requests and re-initialize the
devices, and thus there is no device state to save and restore. After
issuing migrate-incoming, the caller must issue a system_wakeup command to
resume.
In cpr-exec mode, the migrate command saves state to a file and directly
exec's a new version of qemu on the same host, replacing the original process
while retaining its PID. The caller must specify a migration URI that writes
to and reads from a file, and resumes execution via the migrate-incoming
command. Arguments for the new qemu process are taken from the cpr-exec-args
migration parameter, and must include the -incoming option.
Guest RAM must be backed by a memory backend with share=on, but cannot be
memory-backend-ram. The memory is re-mmap'd in the updated process, so guest
ram is efficiently preserved in place, albeit with new virtual addresses.
In addition, the '-migrate-mode-enable cpr-exec' option is required. This
causes secondary guest ram blocks (those not specified on the command line)
to be allocated by mmap'ing a memfd. The memfds are kept open across exec,
their values are saved in special cpr state which is retrieved after exec,
and they are re-mmap'd. Since guest RAM is not copied, and storage blocks
are not migrated, the caller must disable all capabilities related to page
and block
Re: [PATCH V9 00/46] Live Update
On 5/30/2023 9:38 AM, Philippe Mathieu-Daudé wrote: > Hi Steve, > > On 7/2/23 19:44, Steven Sistare wrote: >> To make forward progress on this series and reduce its size, I will be >> posting >> those of its patches that can be independently integrated and have some value >> on their own, to a reduced distribution of reviewers for each. This is what >> I plan to break out: >> >> migration: fix populate_vfio_info >> >> memory: RAM_NAMED_FILE flag >> >> memory: flat section iterator >> >> oslib: qemu_clear_cloexec >> >> migration: simplify blockers >> >> migration: simplify notifiers >> >> python/machine: QEMUMachine full_args >> >> python/machine: QEMUMachine reopen_qmp_connection >> >> qapi: strList_from_string >> qapi: QAPI_LIST_LENGTH >> qapi: strv_from_strList >> qapi: strList unit tests > > The break out looks good, but I guess it is a bit obsolete (as of > today), you probably already re-posted some of these patches as > separate series. Do you mind replying with a "ping" to the posted > ones which still applies and need to be reviewed? Yes, I re-posted this list of patches, some of which have been accepted or are close. I will reply to the threads that are still open and cc you, but I will first verify that they still apply cleanly, and update them if not. Thanks! - Steve
Re: [PATCH V9 00/46] Live Update
Hi Steve, On 7/2/23 19:44, Steven Sistare wrote: To make forward progress on this series and reduce its size, I will be posting those of its patches that can be independently integrated and have some value on their own, to a reduced distribution of reviewers for each. This is what I plan to break out: migration: fix populate_vfio_info memory: RAM_NAMED_FILE flag memory: flat section iterator oslib: qemu_clear_cloexec migration: simplify blockers migration: simplify notifiers python/machine: QEMUMachine full_args python/machine: QEMUMachine reopen_qmp_connection qapi: strList_from_string qapi: QAPI_LIST_LENGTH qapi: strv_from_strList qapi: strList unit tests The break out looks good, but I guess it is a bit obsolete (as of today), you probably already re-posted some of these patches as separate series. Do you mind replying with a "ping" to the posted ones which still applies and need to be reviewed? Thanks, Phil. - Steve On 12/7/2022 10:48 AM, Steven Sistare wrote: This series desperately needs review in its intersection with live migration. The code in other areas has been reviewed and revised multiple times -- thank you!
Re: [PATCH V9 00/46] Live Update
Greetings, For what its worth, our team has been aggressively testing this patch series and to date have not found any deficiencies or bottlenecks whatsoever. Overall, with a very well-tuned system (and linux kernel), we are getting live update downtimes just above 15 seconds, and the vast majority of that downtime is kind of wasted time during kexec (particularly with PCI device probing, if anybody has any tips on that) and has nothing to do whatsoever with the QEMU-side of things. (We use this patchset in a PMEM-based configuration which has been working out extremely well so far). We also tested "back-to-back" live updates as you would expect to do in production. Overall we've done thousands of live updates over the past few months on many different types of hardware without failure. Steven has been really responsive in answering some of our usability questions and we were able to fix those issues. We will continue our testing throughout the year with more heavily-loaded workloads, but all in all we would very much be interested in seeing further reviews on this patch series from others. * *--- Tested-by: Michael Galaxy On 12/7/22 09:48, Steven Sistare wrote: This series desperately needs review in its intersection with live migration. The code in other areas has been reviewed and revised multiple times -- thank you! David, Juan, can you spare some time to review this? I have done my best to order the patches logically (see the labelled groups in this email), and to provide complete and clear cover letter and commit messages. Can I do anything to facilitate, like doing a code walk through via zoom? And of course, I welcome anyone's feedback. Here is the original posting. https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$ - Steve On 7/26/2022 12:09 PM, Steve Sistare wrote: This version of the live update patch series integrates live update into the live migration framework. The new interfaces are: * mode (migration parameter) * cpr-exec-args (migration parameter) * file (migration URI) * migrate-mode-enable (command-line argument) * only-cpr-capable (command-line argument) Provide the cpr-exec and cpr-reboot migration modes for live update. These save and restore VM state, with minimal guest pause time, so that qemu may be updated to a new version in between. The caller sets the mode parameter before invoking the migrate or migrate-incoming commands. In cpr-reboot mode, the migrate command saves state to a file, allowing one to quit qemu, reboot to an updated kernel, start an updated version of qemu, and resume via the migrate-incoming command. The caller must specify a migration URI that writes to and reads from a file. Unlike normal mode, the use of certain local storage options does not block the migration, but the caller must not modify guest block devices between the quit and restart. The guest RAM memory-backend must be shared, and the @x-ignore-shared migration capability must be set, to avoid saving it to the file. Guest RAM must be non-volatile across reboot, which can be achieved by backing it with a dax device, or /dev/shm PKRAM as proposed in https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$ but this is not enforced. The restarted qemu arguments must match those used to initially start qemu, plus the -incoming option. The reboot mode supports vfio devices if the caller first suspends the guest, such as by issuing guest-suspend-ram to the qemu guest agent. The guest drivers' suspend methods flush outstanding requests and re-initialize the devices, and thus there is no device state to save and restore. After issuing migrate-incoming, the caller must issue a system_wakeup command to resume. In cpr-exec mode, the migrate command saves state to a file and directly exec's a new version of qemu on the same host, replacing the original process while retaining its PID. The caller must specify a migration URI that writes to and reads from a file, and resumes execution via the migrate-incoming command. Arguments for the new qemu process are taken from the cpr-exec-args migration parameter, and must include the -incoming option. Guest RAM must be backed by a memory backend with share=on, but cannot be memory-backend-ram. The memory is re-mmap'd in the updated process, so guest ram is efficiently preserved in place, albeit with new virtual addresses. In addition, the '-migrate-mode-enable cpr-exec' option is required. This causes secondary guest ram blocks (those not specified on the command line) to be allocated by mmap'ing a memfd. The memfds are kept open across exec, their values
Re: [PATCH V9 00/46] Live Update
Hey Steven, Have you done any "back-to-back" live update testing before? I am still doing extensive testing on this myself. I am running into a bug that I have not yet diagnosed. It involves the following: 1. Perform a live update (I'm using kexec + PMEM-based live updates). => VM comes back just fine. All is well. 2. Using the *same* QEMU instance: Try to perform another live update agaig, this results in the following error: monitor> migrate_incoming file:qemu.sav qemu-system-x86_64: check_section_footer: Read section footer failed: -5 qemu-system-x86_64: load of migration failed: Invalid argument I'm going to start diving into the code soon, but just thought I would ask first. - Michael On 12/7/22 09:48, Steven Sistare wrote: This series desperately needs review in its intersection with live migration. The code in other areas has been reviewed and revised multiple times -- thank you! David, Juan, can you spare some time to review this? I have done my best to order the patches logically (see the labelled groups in this email), and to provide complete and clear cover letter and commit messages. Can I do anything to facilitate, like doing a code walk through via zoom? And of course, I welcome anyone's feedback. Here is the original posting. https://urldefense.com/v3/__https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3EAFRUVu$ - Steve On 7/26/2022 12:09 PM, Steve Sistare wrote: This version of the live update patch series integrates live update into the live migration framework. The new interfaces are: * mode (migration parameter) * cpr-exec-args (migration parameter) * file (migration URI) * migrate-mode-enable (command-line argument) * only-cpr-capable (command-line argument) Provide the cpr-exec and cpr-reboot migration modes for live update. These save and restore VM state, with minimal guest pause time, so that qemu may be updated to a new version in between. The caller sets the mode parameter before invoking the migrate or migrate-incoming commands. In cpr-reboot mode, the migrate command saves state to a file, allowing one to quit qemu, reboot to an updated kernel, start an updated version of qemu, and resume via the migrate-incoming command. The caller must specify a migration URI that writes to and reads from a file. Unlike normal mode, the use of certain local storage options does not block the migration, but the caller must not modify guest block devices between the quit and restart. The guest RAM memory-backend must be shared, and the @x-ignore-shared migration capability must be set, to avoid saving it to the file. Guest RAM must be non-volatile across reboot, which can be achieved by backing it with a dax device, or /dev/shm PKRAM as proposed in https://urldefense.com/v3/__https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yznaga@oracle.com__;!!GjvTz_vk!U6U0yYHuO4GRyGQUNpn0TQdwlC2QjJfgYC0yE249AuONq8-5rs48pZ6l0K-LOSRHMA9cU2op2U8GC9hU3AKRJQux$ but this is not enforced. The restarted qemu arguments must match those used to initially start qemu, plus the -incoming option. The reboot mode supports vfio devices if the caller first suspends the guest, such as by issuing guest-suspend-ram to the qemu guest agent. The guest drivers' suspend methods flush outstanding requests and re-initialize the devices, and thus there is no device state to save and restore. After issuing migrate-incoming, the caller must issue a system_wakeup command to resume. In cpr-exec mode, the migrate command saves state to a file and directly exec's a new version of qemu on the same host, replacing the original process while retaining its PID. The caller must specify a migration URI that writes to and reads from a file, and resumes execution via the migrate-incoming command. Arguments for the new qemu process are taken from the cpr-exec-args migration parameter, and must include the -incoming option. Guest RAM must be backed by a memory backend with share=on, but cannot be memory-backend-ram. The memory is re-mmap'd in the updated process, so guest ram is efficiently preserved in place, albeit with new virtual addresses. In addition, the '-migrate-mode-enable cpr-exec' option is required. This causes secondary guest ram blocks (those not specified on the command line) to be allocated by mmap'ing a memfd. The memfds are kept open across exec, their values are saved in special cpr state which is retrieved after exec, and they are re-mmap'd. Since guest RAM is not copied, and storage blocks are not migrated, the caller must disable all capabilities related to page and block copy. The implementation ignores all related parameters. The exec mode supports vfio devices by preserving the vfio container, group, device, and event descriptors across the qemu re-exec, and by updating DMA mapping virtual addresses
Re: [PATCH V9 00/46] Live Update
and of course I immediately screwed up and forgot to delete series numbers from the email titles. I will resend. - Steve On 2/7/2023 1:44 PM, Steven Sistare wrote: > To make forward progress on this series and reduce its size, I will be posting > those of its patches that can be independently integrated and have some value > on their own, to a reduced distribution of reviewers for each. This is what > I plan to break out: > > migration: fix populate_vfio_info > > memory: RAM_NAMED_FILE flag > > memory: flat section iterator > > oslib: qemu_clear_cloexec > > migration: simplify blockers > > migration: simplify notifiers > > python/machine: QEMUMachine full_args > > python/machine: QEMUMachine reopen_qmp_connection > > qapi: strList_from_string > qapi: QAPI_LIST_LENGTH > qapi: strv_from_strList > qapi: strList unit tests > > - Steve > > On 12/7/2022 10:48 AM, Steven Sistare wrote: >> This series desperately needs review in its intersection with live migration. >> The code in other areas has been reviewed and revised multiple times -- >> thank you! >> >> David, Juan, can you spare some time to review this? I have done my best to >> order >> the patches logically (see the labelled groups in this email), and to >> provide >> complete and clear cover letter and commit messages. Can I do anything to >> facilitate, >> like doing a code walk through via zoom? >> >> And of course, I welcome anyone's feedback. >> >> Here is the original posting. >> >> https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/ >> >> - Steve >> >> On 7/26/2022 12:09 PM, Steve Sistare wrote: >>> This version of the live update patch series integrates live update into the >>> live migration framework. The new interfaces are: >>> * mode (migration parameter) >>> * cpr-exec-args (migration parameter) >>> * file (migration URI) >>> * migrate-mode-enable (command-line argument) >>> * only-cpr-capable (command-line argument) >>> >>> Provide the cpr-exec and cpr-reboot migration modes for live update. These >>> save and restore VM state, with minimal guest pause time, so that qemu may >>> be >>> updated to a new version in between. The caller sets the mode parameter >>> before invoking the migrate or migrate-incoming commands. >>> >>> In cpr-reboot mode, the migrate command saves state to a file, allowing >>> one to quit qemu, reboot to an updated kernel, start an updated version of >>> qemu, and resume via the migrate-incoming command. The caller must specify >>> a migration URI that writes to and reads from a file. Unlike normal mode, >>> the use of certain local storage options does not block the migration, but >>> the caller must not modify guest block devices between the quit and restart. >>> The guest RAM memory-backend must be shared, and the @x-ignore-shared >>> migration capability must be set, to avoid saving it to the file. Guest RAM >>> must be non-volatile across reboot, which can be achieved by backing it with >>> a dax device, or /dev/shm PKRAM as proposed in >>> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yzn...@oracle.com >>> but this is not enforced. The restarted qemu arguments must match those >>> used >>> to initially start qemu, plus the -incoming option. >>> >>> The reboot mode supports vfio devices if the caller first suspends the >>> guest, >>> such as by issuing guest-suspend-ram to the qemu guest agent. The guest >>> drivers' suspend methods flush outstanding requests and re-initialize the >>> devices, and thus there is no device state to save and restore. After >>> issuing migrate-incoming, the caller must issue a system_wakeup command to >>> resume. >>> >>> In cpr-exec mode, the migrate command saves state to a file and directly >>> exec's a new version of qemu on the same host, replacing the original >>> process >>> while retaining its PID. The caller must specify a migration URI that >>> writes >>> to and reads from a file, and resumes execution via the migrate-incoming >>> command. Arguments for the new qemu process are taken from the >>> cpr-exec-args >>> migration parameter, and must include the -incoming option. >>> >>> Guest RAM must be backed by a memory backend with share=on, but cannot be >>> memory-backend-ram. The memory is re-mmap'd in the updated process, so >>> guest >>> ram is efficiently preserved in place, albeit with new virtual addresses. >>> In addition, the '-migrate-mode-enable cpr-exec' option is required. This >>> causes secondary guest ram blocks (those not specified on the command line) >>> to be allocated by mmap'ing a memfd. The memfds are kept open across exec, >>> their values are saved in special cpr state which is retrieved after exec, >>> and they are re-mmap'd. Since guest RAM is not copied, and storage blocks >>> are not migrated, the caller must disable all capabilities related to page >>> and block copy. The implementation ignores all related parameters. >>>
Re: [PATCH V9 00/46] Live Update
To make forward progress on this series and reduce its size, I will be posting those of its patches that can be independently integrated and have some value on their own, to a reduced distribution of reviewers for each. This is what I plan to break out: migration: fix populate_vfio_info memory: RAM_NAMED_FILE flag memory: flat section iterator oslib: qemu_clear_cloexec migration: simplify blockers migration: simplify notifiers python/machine: QEMUMachine full_args python/machine: QEMUMachine reopen_qmp_connection qapi: strList_from_string qapi: QAPI_LIST_LENGTH qapi: strv_from_strList qapi: strList unit tests - Steve On 12/7/2022 10:48 AM, Steven Sistare wrote: > This series desperately needs review in its intersection with live migration. > The code in other areas has been reviewed and revised multiple times -- thank > you! > > David, Juan, can you spare some time to review this? I have done my best to > order > the patches logically (see the labelled groups in this email), and to provide > complete and clear cover letter and commit messages. Can I do anything to > facilitate, > like doing a code walk through via zoom? > > And of course, I welcome anyone's feedback. > > Here is the original posting. > > https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/ > > - Steve > > On 7/26/2022 12:09 PM, Steve Sistare wrote: >> This version of the live update patch series integrates live update into the >> live migration framework. The new interfaces are: >> * mode (migration parameter) >> * cpr-exec-args (migration parameter) >> * file (migration URI) >> * migrate-mode-enable (command-line argument) >> * only-cpr-capable (command-line argument) >> >> Provide the cpr-exec and cpr-reboot migration modes for live update. These >> save and restore VM state, with minimal guest pause time, so that qemu may be >> updated to a new version in between. The caller sets the mode parameter >> before invoking the migrate or migrate-incoming commands. >> >> In cpr-reboot mode, the migrate command saves state to a file, allowing >> one to quit qemu, reboot to an updated kernel, start an updated version of >> qemu, and resume via the migrate-incoming command. The caller must specify >> a migration URI that writes to and reads from a file. Unlike normal mode, >> the use of certain local storage options does not block the migration, but >> the caller must not modify guest block devices between the quit and restart. >> The guest RAM memory-backend must be shared, and the @x-ignore-shared >> migration capability must be set, to avoid saving it to the file. Guest RAM >> must be non-volatile across reboot, which can be achieved by backing it with >> a dax device, or /dev/shm PKRAM as proposed in >> https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yzn...@oracle.com >> but this is not enforced. The restarted qemu arguments must match those used >> to initially start qemu, plus the -incoming option. >> >> The reboot mode supports vfio devices if the caller first suspends the guest, >> such as by issuing guest-suspend-ram to the qemu guest agent. The guest >> drivers' suspend methods flush outstanding requests and re-initialize the >> devices, and thus there is no device state to save and restore. After >> issuing migrate-incoming, the caller must issue a system_wakeup command to >> resume. >> >> In cpr-exec mode, the migrate command saves state to a file and directly >> exec's a new version of qemu on the same host, replacing the original process >> while retaining its PID. The caller must specify a migration URI that writes >> to and reads from a file, and resumes execution via the migrate-incoming >> command. Arguments for the new qemu process are taken from the cpr-exec-args >> migration parameter, and must include the -incoming option. >> >> Guest RAM must be backed by a memory backend with share=on, but cannot be >> memory-backend-ram. The memory is re-mmap'd in the updated process, so guest >> ram is efficiently preserved in place, albeit with new virtual addresses. >> In addition, the '-migrate-mode-enable cpr-exec' option is required. This >> causes secondary guest ram blocks (those not specified on the command line) >> to be allocated by mmap'ing a memfd. The memfds are kept open across exec, >> their values are saved in special cpr state which is retrieved after exec, >> and they are re-mmap'd. Since guest RAM is not copied, and storage blocks >> are not migrated, the caller must disable all capabilities related to page >> and block copy. The implementation ignores all related parameters. >> >> The exec mode supports vfio devices by preserving the vfio container, group, >> device, and event descriptors across the qemu re-exec, and by updating DMA >> mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and >> VFIO_DMA_MAP_FLAG_VADDR as defined in >> >>
Re: [PATCH V9 00/46] Live Update
This series desperately needs review in its intersection with live migration. The code in other areas has been reviewed and revised multiple times -- thank you! David, Juan, can you spare some time to review this? I have done my best to order the patches logically (see the labelled groups in this email), and to provide complete and clear cover letter and commit messages. Can I do anything to facilitate, like doing a code walk through via zoom? And of course, I welcome anyone's feedback. Here is the original posting. https://lore.kernel.org/qemu-devel/1658851843-236870-1-git-send-email-steven.sist...@oracle.com/ - Steve On 7/26/2022 12:09 PM, Steve Sistare wrote: > This version of the live update patch series integrates live update into the > live migration framework. The new interfaces are: > * mode (migration parameter) > * cpr-exec-args (migration parameter) > * file (migration URI) > * migrate-mode-enable (command-line argument) > * only-cpr-capable (command-line argument) > > Provide the cpr-exec and cpr-reboot migration modes for live update. These > save and restore VM state, with minimal guest pause time, so that qemu may be > updated to a new version in between. The caller sets the mode parameter > before invoking the migrate or migrate-incoming commands. > > In cpr-reboot mode, the migrate command saves state to a file, allowing > one to quit qemu, reboot to an updated kernel, start an updated version of > qemu, and resume via the migrate-incoming command. The caller must specify > a migration URI that writes to and reads from a file. Unlike normal mode, > the use of certain local storage options does not block the migration, but > the caller must not modify guest block devices between the quit and restart. > The guest RAM memory-backend must be shared, and the @x-ignore-shared > migration capability must be set, to avoid saving it to the file. Guest RAM > must be non-volatile across reboot, which can be achieved by backing it with > a dax device, or /dev/shm PKRAM as proposed in > https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yzn...@oracle.com > but this is not enforced. The restarted qemu arguments must match those used > to initially start qemu, plus the -incoming option. > > The reboot mode supports vfio devices if the caller first suspends the guest, > such as by issuing guest-suspend-ram to the qemu guest agent. The guest > drivers' suspend methods flush outstanding requests and re-initialize the > devices, and thus there is no device state to save and restore. After > issuing migrate-incoming, the caller must issue a system_wakeup command to > resume. > > In cpr-exec mode, the migrate command saves state to a file and directly > exec's a new version of qemu on the same host, replacing the original process > while retaining its PID. The caller must specify a migration URI that writes > to and reads from a file, and resumes execution via the migrate-incoming > command. Arguments for the new qemu process are taken from the cpr-exec-args > migration parameter, and must include the -incoming option. > > Guest RAM must be backed by a memory backend with share=on, but cannot be > memory-backend-ram. The memory is re-mmap'd in the updated process, so guest > ram is efficiently preserved in place, albeit with new virtual addresses. > In addition, the '-migrate-mode-enable cpr-exec' option is required. This > causes secondary guest ram blocks (those not specified on the command line) > to be allocated by mmap'ing a memfd. The memfds are kept open across exec, > their values are saved in special cpr state which is retrieved after exec, > and they are re-mmap'd. Since guest RAM is not copied, and storage blocks > are not migrated, the caller must disable all capabilities related to page > and block copy. The implementation ignores all related parameters. > > The exec mode supports vfio devices by preserving the vfio container, group, > device, and event descriptors across the qemu re-exec, and by updating DMA > mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and > VFIO_DMA_MAP_FLAG_VADDR as defined in > > https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sist...@oracle.com > and integrated in Linux kernel 5.12. > > Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode. > The software update is performed while the guest is running to minimize > downtime. > > window 1| window 2 > | > # qemu-system-$arch ... | > -migrate-mode-enable cpr-exec | > QEMU 7.0.50 monitor - type 'help' ... | > (qemu) info status | > VM status: running | > | # yum update qemu > (qemu) migrate_set_parameter mode cpr-exec | > (qemu) migrate_set_parameter
[PATCH V9 00/46] Live Update
This version of the live update patch series integrates live update into the live migration framework. The new interfaces are: * mode (migration parameter) * cpr-exec-args (migration parameter) * file (migration URI) * migrate-mode-enable (command-line argument) * only-cpr-capable (command-line argument) Provide the cpr-exec and cpr-reboot migration modes for live update. These save and restore VM state, with minimal guest pause time, so that qemu may be updated to a new version in between. The caller sets the mode parameter before invoking the migrate or migrate-incoming commands. In cpr-reboot mode, the migrate command saves state to a file, allowing one to quit qemu, reboot to an updated kernel, start an updated version of qemu, and resume via the migrate-incoming command. The caller must specify a migration URI that writes to and reads from a file. Unlike normal mode, the use of certain local storage options does not block the migration, but the caller must not modify guest block devices between the quit and restart. The guest RAM memory-backend must be shared, and the @x-ignore-shared migration capability must be set, to avoid saving it to the file. Guest RAM must be non-volatile across reboot, which can be achieved by backing it with a dax device, or /dev/shm PKRAM as proposed in https://lore.kernel.org/lkml/1617140178-8773-1-git-send-email-anthony.yzn...@oracle.com but this is not enforced. The restarted qemu arguments must match those used to initially start qemu, plus the -incoming option. The reboot mode supports vfio devices if the caller first suspends the guest, such as by issuing guest-suspend-ram to the qemu guest agent. The guest drivers' suspend methods flush outstanding requests and re-initialize the devices, and thus there is no device state to save and restore. After issuing migrate-incoming, the caller must issue a system_wakeup command to resume. In cpr-exec mode, the migrate command saves state to a file and directly exec's a new version of qemu on the same host, replacing the original process while retaining its PID. The caller must specify a migration URI that writes to and reads from a file, and resumes execution via the migrate-incoming command. Arguments for the new qemu process are taken from the cpr-exec-args migration parameter, and must include the -incoming option. Guest RAM must be backed by a memory backend with share=on, but cannot be memory-backend-ram. The memory is re-mmap'd in the updated process, so guest ram is efficiently preserved in place, albeit with new virtual addresses. In addition, the '-migrate-mode-enable cpr-exec' option is required. This causes secondary guest ram blocks (those not specified on the command line) to be allocated by mmap'ing a memfd. The memfds are kept open across exec, their values are saved in special cpr state which is retrieved after exec, and they are re-mmap'd. Since guest RAM is not copied, and storage blocks are not migrated, the caller must disable all capabilities related to page and block copy. The implementation ignores all related parameters. The exec mode supports vfio devices by preserving the vfio container, group, device, and event descriptors across the qemu re-exec, and by updating DMA mapping virtual addresses using VFIO_DMA_UNMAP_FLAG_VADDR and VFIO_DMA_MAP_FLAG_VADDR as defined in https://lore.kernel.org/kvm/1611939252-7240-1-git-send-email-steven.sist...@oracle.com and integrated in Linux kernel 5.12. Here is an example of updating qemu from v7.0.50 to v7.0.51 using exec mode. The software update is performed while the guest is running to minimize downtime. window 1| window 2 | # qemu-system-$arch ... | -migrate-mode-enable cpr-exec | QEMU 7.0.50 monitor - type 'help' ... | (qemu) info status | VM status: running | | # yum update qemu (qemu) migrate_set_parameter mode cpr-exec | (qemu) migrate_set_parameter cpr-exec-args | qemu-system-$arch ... -incoming defer | (qemu) migrate -d file:/tmp/qemu.sav| QEMU 7.0.51 monitor - type 'help' ... | (qemu) info status | VM status: paused (inmigrate) | (qemu) migrate_incoming file:/tmp/qemu.sav | (qemu) info status | VM status: running | Here is an example of updating the host kernel using reboot mode. window 1| window 2 | # qemu-system-$arch ... mem-path=/dev/dax0.0| -migrate-mode-enable cpr-reboot | QEMU 7.0.50 monitor - type 'help' ... | (qemu) info status | VM status: running
