RE: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
If user executes the shutdown normally and QEMU crashes, I think this is unacceptable. Since we can avoid this situation, why not do it? Thanks, Lei. -Original Message- From: Lukas Straub Sent: Sunday, February 14, 2021 7:46 PM To: Rao, Lei Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; jasow...@redhat.com; zhang.zhanghaili...@huawei.com; quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint On Fri, 29 Jan 2021 02:57:57 + "Rao, Lei" wrote: > The state will be set RUN_STATE_COLO in colo_do_checkpoint_transaction(). If > the guest executes power off or shutdown at this time and the QEMU main > thread will call vm_shutdown(), it will set the state to RUN_STATE_SHUTDOWN. > The state switch from RUN_STATE_COLO to RUN_STATE_SHUTDOWN is not defined in > runstate_transitions_def. this will cause QEMU crash. Although this is small > probability, it may still happen. This patch fixes the 'colo' -> 'shutdown' transition. AFAIK then colo_do_checkpoint_transaction will call vm_start() again, which does 'shutdown' -> 'running' and (rightfully) crashes. So I think it is better to crash here too. > By the way. Do you have any comments about other patches? > Thanks, > Lei. > > -Original Message- > From: Lukas Straub > Sent: Thursday, January 28, 2021 2:24 AM > To: Rao, Lei > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; > quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown > during checkpoint > > On Thu, 21 Jan 2021 01:48:31 + > "Rao, Lei" wrote: > > > The Primary VM can be shut down when it is in COLO state, which may trigger > > this bug. > > Do you have a backtrace for this bug? > > > About 'shutdown' -> 'colo' -> 'running', I think you are right, I did have > > the problems you said. For 'shutdown'->'colo', The fixed > > patch(5647051f432b7c9b57525470b0a79a31339062d2) have been merged. > > Recently, I found another bug as follows in the test. > > qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'running' > > Aborted (core dumped) > > The gdb bt as following: > > #0 __GI_raise (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:50 > > #1 0x7faa3d613859 in __GI_abort () at abort.c:79 > > #2 0x55c5a21268fd in runstate_set (new_state=RUN_STATE_RUNNING) at > > vl.c:723 > > #3 0x55c5a1f8cae4 in vm_prepare_start () at > > /home/workspace/colo-qemu/cpus.c:2206 > > #4 0x55c5a1f8cb1b in vm_start () at > > /home/workspace/colo-qemu/cpus.c:2213 > > #5 0x55c5a2332bba in migration_iteration_finish (s=0x55c5a4658810) > > at migration/migration.c:3376 > > #6 0x55c5a2332f3b in migration_thread (opaque=0x55c5a4658810) at > > migration/migration.c:3527 > > #7 0x55c5a251d68a in qemu_thread_start (args=0x55c5a5491a70) at > > util/qemu-thread-posix.c:519 > > #8 0x7faa3d7e9609 in start_thread (arg=) at > > pthread_create.c:477 > > #9 0x7faa3d710293 in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > > > For the bug, I made the following changes: > > @@ -3379,7 +3379,9 @@ static void > > migration_iteration_finish(MigrationState *s) > > case MIGRATION_STATUS_CANCELLED: > > case MIGRATION_STATUS_CANCELLING: > > if (s->vm_was_running) { > > -vm_start(); > > +if (!runstate_check(RUN_STATE_SHUTDOWN)) { > > +vm_start(); > > +} > > } else { > > if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { > > runstate_set(RUN_STATE_POSTMIGRATE); > > > > I will send the patch to community after more test. > > > > Thanks, > > Lei. > > > > -Original Message- > > From: Lukas Straub > > Sent: Thursday, January 21, 2021 3:13 AM > > To: Rao, Lei > > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; > > quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org > > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown > > during checkpoint > > > > On Wed, 13 Jan 2021 10:46:27 +0800 > > leirao wrote: > > > > > From: "Rao, Lei" > > > > > > This patch fixes the following: > > > qemu-syst
Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
On Fri, 29 Jan 2021 02:57:57 + "Rao, Lei" wrote: > The state will be set RUN_STATE_COLO in colo_do_checkpoint_transaction(). If > the guest executes power off or shutdown at this time and the QEMU main > thread will call vm_shutdown(), it will set the state to RUN_STATE_SHUTDOWN. > The state switch from RUN_STATE_COLO to RUN_STATE_SHUTDOWN is not defined in > runstate_transitions_def. this will cause QEMU crash. Although this is small > probability, it may still happen. This patch fixes the 'colo' -> 'shutdown' transition. AFAIK then colo_do_checkpoint_transaction will call vm_start() again, which does 'shutdown' -> 'running' and (rightfully) crashes. So I think it is better to crash here too. > By the way. Do you have any comments about other patches? > Thanks, > Lei. > > -Original Message- > From: Lukas Straub > Sent: Thursday, January 28, 2021 2:24 AM > To: Rao, Lei > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; quint...@redhat.com; > dgilb...@redhat.com; qemu-devel@nongnu.org > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during > checkpoint > > On Thu, 21 Jan 2021 01:48:31 + > "Rao, Lei" wrote: > > > The Primary VM can be shut down when it is in COLO state, which may trigger > > this bug. > > Do you have a backtrace for this bug? > > > About 'shutdown' -> 'colo' -> 'running', I think you are right, I did have > > the problems you said. For 'shutdown'->'colo', The fixed > > patch(5647051f432b7c9b57525470b0a79a31339062d2) have been merged. > > Recently, I found another bug as follows in the test. > > qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'running' > > Aborted (core dumped) > > The gdb bt as following: > > #0 __GI_raise (sig=sig@entry=6) at > > ../sysdeps/unix/sysv/linux/raise.c:50 > > #1 0x7faa3d613859 in __GI_abort () at abort.c:79 > > #2 0x55c5a21268fd in runstate_set (new_state=RUN_STATE_RUNNING) at > > vl.c:723 > > #3 0x55c5a1f8cae4 in vm_prepare_start () at > > /home/workspace/colo-qemu/cpus.c:2206 > > #4 0x55c5a1f8cb1b in vm_start () at > > /home/workspace/colo-qemu/cpus.c:2213 > > #5 0x55c5a2332bba in migration_iteration_finish (s=0x55c5a4658810) > > at migration/migration.c:3376 > > #6 0x55c5a2332f3b in migration_thread (opaque=0x55c5a4658810) at > > migration/migration.c:3527 > > #7 0x55c5a251d68a in qemu_thread_start (args=0x55c5a5491a70) at > > util/qemu-thread-posix.c:519 > > #8 0x7faa3d7e9609 in start_thread (arg=) at > > pthread_create.c:477 > > #9 0x7faa3d710293 in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > > > For the bug, I made the following changes: > > @@ -3379,7 +3379,9 @@ static void > > migration_iteration_finish(MigrationState *s) > > case MIGRATION_STATUS_CANCELLED: > > case MIGRATION_STATUS_CANCELLING: > > if (s->vm_was_running) { > > -vm_start(); > > +if (!runstate_check(RUN_STATE_SHUTDOWN)) { > > +vm_start(); > > +} > > } else { > > if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { > > runstate_set(RUN_STATE_POSTMIGRATE); > > > > I will send the patch to community after more test. > > > > Thanks, > > Lei. > > > > -Original Message- > > From: Lukas Straub > > Sent: Thursday, January 21, 2021 3:13 AM > > To: Rao, Lei > > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; > > quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org > > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown > > during checkpoint > > > > On Wed, 13 Jan 2021 10:46:27 +0800 > > leirao wrote: > > > > > From: "Rao, Lei" > > > > > > This patch fixes the following: > > > qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown' > > > Aborted (core dumped) > > > > > > Signed-off-by: Lei Rao > > > > I wonder how that is possible, since the VM is stopped during 'colo' state. > > > > Unrelated to this patch, I think this area needs some work since the > > following unintended runstate transition is possible: > > 'shutdown' -> 'colo' -> 'running'. > > > > >
RE: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
The state will be set RUN_STATE_COLO in colo_do_checkpoint_transaction(). If the guest executes power off or shutdown at this time and the QEMU main thread will call vm_shutdown(), it will set the state to RUN_STATE_SHUTDOWN. The state switch from RUN_STATE_COLO to RUN_STATE_SHUTDOWN is not defined in runstate_transitions_def. this will cause QEMU crash. Although this is small probability, it may still happen. By the way. Do you have any comments about other patches? Thanks, Lei. -Original Message- From: Lukas Straub Sent: Thursday, January 28, 2021 2:24 AM To: Rao, Lei Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; jasow...@redhat.com; zhang.zhanghaili...@huawei.com; quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint On Thu, 21 Jan 2021 01:48:31 + "Rao, Lei" wrote: > The Primary VM can be shut down when it is in COLO state, which may trigger > this bug. Do you have a backtrace for this bug? > About 'shutdown' -> 'colo' -> 'running', I think you are right, I did have > the problems you said. For 'shutdown'->'colo', The fixed > patch(5647051f432b7c9b57525470b0a79a31339062d2) have been merged. > Recently, I found another bug as follows in the test. > qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'running' > Aborted (core dumped) > The gdb bt as following: > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > #1 0x7faa3d613859 in __GI_abort () at abort.c:79 > #2 0x55c5a21268fd in runstate_set (new_state=RUN_STATE_RUNNING) at > vl.c:723 > #3 0x55c5a1f8cae4 in vm_prepare_start () at > /home/workspace/colo-qemu/cpus.c:2206 > #4 0x55c5a1f8cb1b in vm_start () at > /home/workspace/colo-qemu/cpus.c:2213 > #5 0x55c5a2332bba in migration_iteration_finish (s=0x55c5a4658810) > at migration/migration.c:3376 > #6 0x55c5a2332f3b in migration_thread (opaque=0x55c5a4658810) at > migration/migration.c:3527 > #7 0x55c5a251d68a in qemu_thread_start (args=0x55c5a5491a70) at > util/qemu-thread-posix.c:519 > #8 0x7faa3d7e9609 in start_thread (arg=) at > pthread_create.c:477 > #9 0x7faa3d710293 in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > For the bug, I made the following changes: > @@ -3379,7 +3379,9 @@ static void > migration_iteration_finish(MigrationState *s) > case MIGRATION_STATUS_CANCELLED: > case MIGRATION_STATUS_CANCELLING: > if (s->vm_was_running) { > -vm_start(); > +if (!runstate_check(RUN_STATE_SHUTDOWN)) { > +vm_start(); > +} > } else { > if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { > runstate_set(RUN_STATE_POSTMIGRATE); > > I will send the patch to community after more test. > > Thanks, > Lei. > > -Original Message- > From: Lukas Straub > Sent: Thursday, January 21, 2021 3:13 AM > To: Rao, Lei > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; > quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown > during checkpoint > > On Wed, 13 Jan 2021 10:46:27 +0800 > leirao wrote: > > > From: "Rao, Lei" > > > > This patch fixes the following: > > qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown' > > Aborted (core dumped) > > > > Signed-off-by: Lei Rao > > I wonder how that is possible, since the VM is stopped during 'colo' state. > > Unrelated to this patch, I think this area needs some work since the > following unintended runstate transition is possible: > 'shutdown' -> 'colo' -> 'running'. > > > --- > > softmmu/runstate.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/softmmu/runstate.c b/softmmu/runstate.c index > > 636aab0..455ad0d 100644 > > --- a/softmmu/runstate.c > > +++ b/softmmu/runstate.c > > @@ -125,6 +125,7 @@ static const RunStateTransition > > runstate_transitions_def[] = { > > { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH }, > > > > { RUN_STATE_COLO, RUN_STATE_RUNNING }, > > +{ RUN_STATE_COLO, RUN_STATE_SHUTDOWN}, > > > > { RUN_STATE_RUNNING, RUN_STATE_DEBUG }, > > { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR }, > > > --
Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
On Thu, 21 Jan 2021 01:48:31 + "Rao, Lei" wrote: > The Primary VM can be shut down when it is in COLO state, which may trigger > this bug. Do you have a backtrace for this bug? > About 'shutdown' -> 'colo' -> 'running', I think you are right, I did have > the problems you said. For 'shutdown'->'colo', The fixed > patch(5647051f432b7c9b57525470b0a79a31339062d2) have been merged. > Recently, I found another bug as follows in the test. > qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'running' > Aborted (core dumped) > The gdb bt as following: > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > #1 0x7faa3d613859 in __GI_abort () at abort.c:79 > #2 0x55c5a21268fd in runstate_set (new_state=RUN_STATE_RUNNING) at > vl.c:723 > #3 0x55c5a1f8cae4 in vm_prepare_start () at > /home/workspace/colo-qemu/cpus.c:2206 > #4 0x55c5a1f8cb1b in vm_start () at > /home/workspace/colo-qemu/cpus.c:2213 > #5 0x55c5a2332bba in migration_iteration_finish (s=0x55c5a4658810) > at migration/migration.c:3376 > #6 0x55c5a2332f3b in migration_thread (opaque=0x55c5a4658810) at > migration/migration.c:3527 > #7 0x55c5a251d68a in qemu_thread_start (args=0x55c5a5491a70) at > util/qemu-thread-posix.c:519 > #8 0x7faa3d7e9609 in start_thread (arg=) at > pthread_create.c:477 > #9 0x7faa3d710293 in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 > > For the bug, I made the following changes: > @@ -3379,7 +3379,9 @@ static void > migration_iteration_finish(MigrationState *s) > case MIGRATION_STATUS_CANCELLED: > case MIGRATION_STATUS_CANCELLING: > if (s->vm_was_running) { > -vm_start(); > +if (!runstate_check(RUN_STATE_SHUTDOWN)) { > +vm_start(); > +} > } else { > if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { > runstate_set(RUN_STATE_POSTMIGRATE); > > I will send the patch to community after more test. > > Thanks, > Lei. > > -Original Message- > From: Lukas Straub > Sent: Thursday, January 21, 2021 3:13 AM > To: Rao, Lei > Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; > jasow...@redhat.com; zhang.zhanghaili...@huawei.com; quint...@redhat.com; > dgilb...@redhat.com; qemu-devel@nongnu.org > Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during > checkpoint > > On Wed, 13 Jan 2021 10:46:27 +0800 > leirao wrote: > > > From: "Rao, Lei" > > > > This patch fixes the following: > > qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown' > > Aborted (core dumped) > > > > Signed-off-by: Lei Rao > > I wonder how that is possible, since the VM is stopped during 'colo' state. > > Unrelated to this patch, I think this area needs some work since the > following unintended runstate transition is possible: > 'shutdown' -> 'colo' -> 'running'. > > > --- > > softmmu/runstate.c | 1 + > > 1 file changed, 1 insertion(+) > > > > diff --git a/softmmu/runstate.c b/softmmu/runstate.c index > > 636aab0..455ad0d 100644 > > --- a/softmmu/runstate.c > > +++ b/softmmu/runstate.c > > @@ -125,6 +125,7 @@ static const RunStateTransition > > runstate_transitions_def[] = { > > { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH }, > > > > { RUN_STATE_COLO, RUN_STATE_RUNNING }, > > +{ RUN_STATE_COLO, RUN_STATE_SHUTDOWN}, > > > > { RUN_STATE_RUNNING, RUN_STATE_DEBUG }, > > { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR }, > > > -- pgpLh3IRx4Bev.pgp Description: OpenPGP digital signature
RE: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
The Primary VM can be shut down when it is in COLO state, which may trigger this bug. About 'shutdown' -> 'colo' -> 'running', I think you are right, I did have the problems you said. For 'shutdown'->'colo', The fixed patch(5647051f432b7c9b57525470b0a79a31339062d2) have been merged. Recently, I found another bug as follows in the test. qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'running' Aborted (core dumped) The gdb bt as following: #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x7faa3d613859 in __GI_abort () at abort.c:79 #2 0x55c5a21268fd in runstate_set (new_state=RUN_STATE_RUNNING) at vl.c:723 #3 0x55c5a1f8cae4 in vm_prepare_start () at /home/workspace/colo-qemu/cpus.c:2206 #4 0x55c5a1f8cb1b in vm_start () at /home/workspace/colo-qemu/cpus.c:2213 #5 0x55c5a2332bba in migration_iteration_finish (s=0x55c5a4658810) at migration/migration.c:3376 #6 0x55c5a2332f3b in migration_thread (opaque=0x55c5a4658810) at migration/migration.c:3527 #7 0x55c5a251d68a in qemu_thread_start (args=0x55c5a5491a70) at util/qemu-thread-posix.c:519 #8 0x7faa3d7e9609 in start_thread (arg=) at pthread_create.c:477 #9 0x7faa3d710293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95 For the bug, I made the following changes: @@ -3379,7 +3379,9 @@ static void migration_iteration_finish(MigrationState *s) case MIGRATION_STATUS_CANCELLED: case MIGRATION_STATUS_CANCELLING: if (s->vm_was_running) { -vm_start(); +if (!runstate_check(RUN_STATE_SHUTDOWN)) { +vm_start(); +} } else { if (runstate_check(RUN_STATE_FINISH_MIGRATE)) { runstate_set(RUN_STATE_POSTMIGRATE); I will send the patch to community after more test. Thanks, Lei. -Original Message- From: Lukas Straub Sent: Thursday, January 21, 2021 3:13 AM To: Rao, Lei Cc: Zhang, Chen ; lizhij...@cn.fujitsu.com; jasow...@redhat.com; zhang.zhanghaili...@huawei.com; quint...@redhat.com; dgilb...@redhat.com; qemu-devel@nongnu.org Subject: Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint On Wed, 13 Jan 2021 10:46:27 +0800 leirao wrote: > From: "Rao, Lei" > > This patch fixes the following: > qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown' > Aborted (core dumped) > > Signed-off-by: Lei Rao I wonder how that is possible, since the VM is stopped during 'colo' state. Unrelated to this patch, I think this area needs some work since the following unintended runstate transition is possible: 'shutdown' -> 'colo' -> 'running'. > --- > softmmu/runstate.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/softmmu/runstate.c b/softmmu/runstate.c index > 636aab0..455ad0d 100644 > --- a/softmmu/runstate.c > +++ b/softmmu/runstate.c > @@ -125,6 +125,7 @@ static const RunStateTransition > runstate_transitions_def[] = { > { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH }, > > { RUN_STATE_COLO, RUN_STATE_RUNNING }, > +{ RUN_STATE_COLO, RUN_STATE_SHUTDOWN}, > > { RUN_STATE_RUNNING, RUN_STATE_DEBUG }, > { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR }, --
Re: [PATCH 02/10] Fix the qemu crash when guest shutdown during checkpoint
On Wed, 13 Jan 2021 10:46:27 +0800 leirao wrote: > From: "Rao, Lei" > > This patch fixes the following: > qemu-system-x86_64: invalid runstate transition: 'colo' ->'shutdown' > Aborted (core dumped) > > Signed-off-by: Lei Rao I wonder how that is possible, since the VM is stopped during 'colo' state. Unrelated to this patch, I think this area needs some work since the following unintended runstate transition is possible: 'shutdown' -> 'colo' -> 'running'. > --- > softmmu/runstate.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/softmmu/runstate.c b/softmmu/runstate.c > index 636aab0..455ad0d 100644 > --- a/softmmu/runstate.c > +++ b/softmmu/runstate.c > @@ -125,6 +125,7 @@ static const RunStateTransition > runstate_transitions_def[] = { > { RUN_STATE_RESTORE_VM, RUN_STATE_PRELAUNCH }, > > { RUN_STATE_COLO, RUN_STATE_RUNNING }, > +{ RUN_STATE_COLO, RUN_STATE_SHUTDOWN}, > > { RUN_STATE_RUNNING, RUN_STATE_DEBUG }, > { RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR }, -- pgpWfUClGnK25.pgp Description: OpenPGP digital signature