Re: [raw] Guest stuck during live live-migration
Hi, We made further tests on this, it seems related to disk fragmentation, I can reproduce on a local-mirroring 5.2.0-rc2(I assume it is the same issue than live-migration, to confirm). I put details here If you can have a look: https://bugs.launchpad.net/qemu/+bug/1912224 Issue is ext4 tools does not consider file as enough fragmented to unfragment, and we cannot live-migrate because of the issue. Alexandre Kevin Wolf wrote on mer. [2020-déc.-02 16:33:47 +0100]: > Am 02.12.2020 um 16:09 hat Quentin Grolleau geschrieben: > > Do you think that, applying this patch ( replacing by "#if 0" there : > > https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ), > > could affect for any reason the customer data ? > > > > As we are on full NVME and 10G networks it should fix our vm which > > completely freeze. > > It's not a real fix for the general case, of course, but yes, you can do > that safely. It means that QEMU will assume that all of your raw images > are fully allocated. > > Kevin > > > De : Qemu-devel > > de la part de > > Quentin Grolleau > > Envoyé : mardi 24 novembre 2020 13:58:53 > > à : Kevin Wolf > > Cc : qemu-devel@nongnu.org; qemu-bl...@nongnu.org > > Objet : RE: [raw] Guest stuck during live live-migration > > > > Thanks Kevin, > > > > > > Hello, > > > > > > > > In our company, we are hosting a large number of Vm, hosted behind > > > > Openstack (so libvirt/qemu). > > > > A large majority of our Vms are runnign with local data only, stored on > > > > NVME, and most of them are RAW disks. > > > > > > > > With Qemu 4.0 (can be even with older version) we see strange > > > > live-migration comportement: > > > > > First of all, 4.0 is relatively old. Generally it is worth retrying with > > > the most recent code (git master or 5.2.0-rc2) before having a closer > > > look at problems, because it is frustrating to spend considerable time > > > debugging an issue and then find out it has already been fixed a year > > > ago. > > > > > I will try to build it the most recent code > > > > > > I will try to build with the most recent code, but it will take me some > > time to do it > > > > > > > > - some Vms live migrate at very high speed without issue (> 6 Gbps) > > > > - some Vms are running correctly, but migrating at a strange low > > > > speed (3Gbps) > > > > - some Vms are migrating at a very low speed (1Gbps, sometime less) > > > > and during the migration the guest is completely I/O stuck > > > > > > > > When this issue happen the VM is completly block, iostat in the Vm show > > > > us a latency of 30 secs > > > > > Can you get the stack backtraces of all QEMU threads while the VM is > > > blocked (e.g. with gdb or pstack)? > > > > (gdb) thread apply all bt > > > > Thread 20 (Thread 0x7f8a0effd700 (LWP 201248)): > > #0 pthread_cond_wait@@GLIBC_2.3.2 () at > > ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > > #1 0x56520139878b in qemu_cond_wait_impl (cond=0x5652020f27b0, > > mutex=0x5652020f27e8, file=0x5652014e4178 > > "/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c", line=214) at > > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:161 > > #2 0x5652012a264d in vnc_worker_thread_loop > > (queue=queue@entry=0x5652020f27b0) at > > /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:214 > > #3 0x5652012a2c18 in vnc_worker_thread (arg=arg@entry=0x5652020f27b0) > > at /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:324 > > #4 0x565201398116 in qemu_thread_start (args=) at > > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 > > #5 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0effd700) at > > pthread_create.c:333 > > #6 0x7f8a5df8041d in clone () at > > ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 > > > > Thread 19 (Thread 0x7f8a0700 (LWP 201222)): > > #0 __lll_lock_wait () at > > ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > > #1 0x7f8a5e24cdbd in __GI___pthread_mutex_lock > > (mutex=mutex@entry=0x565201adb680 ) at > > ../nptl/pthread_mutex_lock.c:80 > > #2 0x565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680 > > , file=0x5652013d7c68 > > "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at > > /root/qemu_debug_LSEEK/qemu_
Re: [raw] Guest stuck during live live-migration
On 11/23/2020 05:36 PM, Quentin Grolleau wrote: Hello, In our company, we are hosting a large number of Vm, hosted behind Openstack (so libvirt/qemu). A large majority of our Vms are runnign with local data only, stored on NVME, and most of them are RAW disks. With Qemu 4.0(can be even with older version)we see strange live-migrationcomportement: - some Vms live migrate at very high speed without issue (> 6 Gbps) - some Vms are running correctly, but migrating at a strange low speed (3Gbps) - some Vms are migrating at a very low speed (1Gbps, sometime less) and during the migration the guest is completely I/O stuck When this issue happen the VM is completly block, iostat in the Vm show us a latency of 30 secs First we thought it was related to an hardware issuewe check it, we comparing different hardware, but no issue where found there So one of my colleague had the idea to limit with "tc" the bandwidth on the interface the migration was done, and it worked the VM didn't lose any ping nor being I/O stuck Important point : Once the Vm have been migrate (with the limitation ) one time, if we migrate it again right after, the migration will be done at full speed (8-9Gb/s) without freezing the Vm It only happen on existing VM, we tried to replicate with a fresh instance with exactly the same spec and nothing was happening We tried to replicate the workload inside the VM but there was no way to replicate the case. So it was not related to the workload nor to the server that hosts the Vm So we thought about the disk of the instance : the raw file. We also tried to strace -c the process during the live-migration and it was doing a lot of "lseek" and we found this : https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00462.html So i rebuilt Qemu with this patch and the live-migration went well, at high speedand with no VM freeze ( https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601) Do you have a way to avoid the "lseek" mechanism as it consumes more resources to find the holes in the disk and don't let any for the VM ? Server hosting the VM : - Bi-Xeon hosts With NVME storage and 10 Go Network card - Qemu 4.0 And Libvirt 5.4 - Kernel 4.18.0.25 Guest having the issue : - raw image with Debian 8 Here the qemu img on the disk : > qemu-img info disk image: disk file format: raw virtual size: 400G (429496729600 bytes) disk size: 400G Could you share the migration options that you use and "info migrate" for both stuck and non-stuck cases? Best, Wei
Re: [raw] Guest stuck during live live-migration
Am 02.12.2020 um 16:09 hat Quentin Grolleau geschrieben: > Do you think that, applying this patch ( replacing by "#if 0" there : > https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ), > could affect for any reason the customer data ? > > As we are on full NVME and 10G networks it should fix our vm which > completely freeze. It's not a real fix for the general case, of course, but yes, you can do that safely. It means that QEMU will assume that all of your raw images are fully allocated. Kevin > De : Qemu-devel > de la part de Quentin Grolleau > Envoyé : mardi 24 novembre 2020 13:58:53 > à : Kevin Wolf > Cc : qemu-devel@nongnu.org; qemu-bl...@nongnu.org > Objet : RE: [raw] Guest stuck during live live-migration > > Thanks Kevin, > > > > Hello, > > > > > > In our company, we are hosting a large number of Vm, hosted behind > > > Openstack (so libvirt/qemu). > > > A large majority of our Vms are runnign with local data only, stored on > > > NVME, and most of them are RAW disks. > > > > > > With Qemu 4.0 (can be even with older version) we see strange > > > live-migration comportement: > > > First of all, 4.0 is relatively old. Generally it is worth retrying with > > the most recent code (git master or 5.2.0-rc2) before having a closer > > look at problems, because it is frustrating to spend considerable time > > debugging an issue and then find out it has already been fixed a year > > ago. > > > I will try to build it the most recent code > > > I will try to build with the most recent code, but it will take me some time > to do it > > > > > - some Vms live migrate at very high speed without issue (> 6 Gbps) > > > - some Vms are running correctly, but migrating at a strange low > > > speed (3Gbps) > > > - some Vms are migrating at a very low speed (1Gbps, sometime less) > > > and during the migration the guest is completely I/O stuck > > > > > > When this issue happen the VM is completly block, iostat in the Vm show > > > us a latency of 30 secs > > > Can you get the stack backtraces of all QEMU threads while the VM is > > blocked (e.g. with gdb or pstack)? > > (gdb) thread apply all bt > > Thread 20 (Thread 0x7f8a0effd700 (LWP 201248)): > #0 pthread_cond_wait@@GLIBC_2.3.2 () at > ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 > #1 0x56520139878b in qemu_cond_wait_impl (cond=0x5652020f27b0, > mutex=0x5652020f27e8, file=0x5652014e4178 > "/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c", line=214) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:161 > #2 0x5652012a264d in vnc_worker_thread_loop > (queue=queue@entry=0x5652020f27b0) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:214 > #3 0x5652012a2c18 in vnc_worker_thread (arg=arg@entry=0x5652020f27b0) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:324 > #4 0x565201398116 in qemu_thread_start (args=) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 > #5 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0effd700) at > pthread_create.c:333 > #6 0x7f8a5df8041d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 > > Thread 19 (Thread 0x7f8a0700 (LWP 201222)): > #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 > #1 0x7f8a5e24cdbd in __GI___pthread_mutex_lock > (mutex=mutex@entry=0x565201adb680 ) at > ../nptl/pthread_mutex_lock.c:80 > #2 0x565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680 > , file=0x5652013d7c68 > "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66 > #3 0x565200f7d00e in qemu_mutex_lock_iothread_impl > (file=file@entry=0x5652013d7c68 > "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", > line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850 > #4 0x565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202354480) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089 > #5 0x565200f7d1ce in qemu_kvm_cpu_thread_fn > (arg=arg@entry=0x565202354480) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281 > #6 0x565201398116 in qemu_thread_start (args=) at > /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 > #7 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0700) at > pthread_create.c:333 > #8 0x7f8a5df8041d in clone () at > ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 > >
RE: [raw] Guest stuck during live live-migration
Do you think that, applying this patch ( replacing by "#if 0" there : https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ), could affect for any reason the customer data ? As we are on full NVME and 10G networks it should fix our vm which completely freeze. Quentin De : Qemu-devel de la part de Quentin Grolleau Envoyé : mardi 24 novembre 2020 13:58:53 À : Kevin Wolf Cc : qemu-devel@nongnu.org; qemu-bl...@nongnu.org Objet : RE: [raw] Guest stuck during live live-migration Thanks Kevin, > > Hello, > > > > In our company, we are hosting a large number of Vm, hosted behind > > Openstack (so libvirt/qemu). > > A large majority of our Vms are runnign with local data only, stored on > > NVME, and most of them are RAW disks. > > > > With Qemu 4.0 (can be even with older version) we see strange > > live-migration comportement: > First of all, 4.0 is relatively old. Generally it is worth retrying with > the most recent code (git master or 5.2.0-rc2) before having a closer > look at problems, because it is frustrating to spend considerable time > debugging an issue and then find out it has already been fixed a year > ago. > I will try to build it the most recent code I will try to build with the most recent code, but it will take me some time to do it > > - some Vms live migrate at very high speed without issue (> 6 Gbps) > > - some Vms are running correctly, but migrating at a strange low speed > > (3Gbps) > > - some Vms are migrating at a very low speed (1Gbps, sometime less) and > > during the migration the guest is completely I/O stuck > > > > When this issue happen the VM is completly block, iostat in the Vm show us > > a latency of 30 secs > Can you get the stack backtraces of all QEMU threads while the VM is > blocked (e.g. with gdb or pstack)? (gdb) thread apply all bt Thread 20 (Thread 0x7f8a0effd700 (LWP 201248)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x56520139878b in qemu_cond_wait_impl (cond=0x5652020f27b0, mutex=0x5652020f27e8, file=0x5652014e4178 "/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c", line=214) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:161 #2 0x5652012a264d in vnc_worker_thread_loop (queue=queue@entry=0x5652020f27b0) at /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:214 #3 0x5652012a2c18 in vnc_worker_thread (arg=arg@entry=0x5652020f27b0) at /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:324 #4 0x565201398116 in qemu_thread_start (args=) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 #5 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0effd700) at pthread_create.c:333 #6 0x7f8a5df8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 19 (Thread 0x7f8a0700 (LWP 201222)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x7f8a5e24cdbd in __GI___pthread_mutex_lock (mutex=mutex@entry=0x565201adb680 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680 , file=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66 #3 0x565200f7d00e in qemu_mutex_lock_iothread_impl (file=file@entry=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850 #4 0x565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202354480) at /root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089 #5 0x565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202354480) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281 #6 0x565201398116 in qemu_thread_start (args=) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 #7 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0700) at pthread_create.c:333 #8 0x7f8a5df8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 18 (Thread 0x7f8a2cff9700 (LWP 201221)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x7f8a5e24cdbd in __GI___pthread_mutex_lock (mutex=mutex@entry=0x565201adb680 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680 , file=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66 #3 0x565200f7d00e in qemu_mutex_lock_iothread_impl (file=file@entry=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=line@entry=208
RE: [raw] Guest stuck during live live-migration
Thanks Kevin, > > Hello, > > > > In our company, we are hosting a large number of Vm, hosted behind > > Openstack (so libvirt/qemu). > > A large majority of our Vms are runnign with local data only, stored on > > NVME, and most of them are RAW disks. > > > > With Qemu 4.0 (can be even with older version) we see strange > > live-migration comportement: > First of all, 4.0 is relatively old. Generally it is worth retrying with > the most recent code (git master or 5.2.0-rc2) before having a closer > look at problems, because it is frustrating to spend considerable time > debugging an issue and then find out it has already been fixed a year > ago. > I will try to build it the most recent code I will try to build with the most recent code, but it will take me some time to do it > > - some Vms live migrate at very high speed without issue (> 6 Gbps) > > - some Vms are running correctly, but migrating at a strange low speed > > (3Gbps) > > - some Vms are migrating at a very low speed (1Gbps, sometime less) and > > during the migration the guest is completely I/O stuck > > > > When this issue happen the VM is completly block, iostat in the Vm show us > > a latency of 30 secs > Can you get the stack backtraces of all QEMU threads while the VM is > blocked (e.g. with gdb or pstack)? (gdb) thread apply all bt Thread 20 (Thread 0x7f8a0effd700 (LWP 201248)): #0 pthread_cond_wait@@GLIBC_2.3.2 () at ../sysdeps/unix/sysv/linux/x86_64/pthread_cond_wait.S:185 #1 0x56520139878b in qemu_cond_wait_impl (cond=0x5652020f27b0, mutex=0x5652020f27e8, file=0x5652014e4178 "/root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c", line=214) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:161 #2 0x5652012a264d in vnc_worker_thread_loop (queue=queue@entry=0x5652020f27b0) at /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:214 #3 0x5652012a2c18 in vnc_worker_thread (arg=arg@entry=0x5652020f27b0) at /root/qemu_debug_LSEEK/qemu_debug/qemu/ui/vnc-jobs.c:324 #4 0x565201398116 in qemu_thread_start (args=) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 #5 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0effd700) at pthread_create.c:333 #6 0x7f8a5df8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 19 (Thread 0x7f8a0700 (LWP 201222)): #0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135 #1 0x7f8a5e24cdbd in __GI___pthread_mutex_lock (mutex=mutex@entry=0x565201adb680 ) at ../nptl/pthread_mutex_lock.c:80 #2 0x565201398263 in qemu_mutex_lock_impl (mutex=0x565201adb680 , file=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:66 #3 0x565200f7d00e in qemu_mutex_lock_iothread_impl (file=file@entry=0x5652013d7c68 "/root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c", line=line@entry=2089) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1850 #4 0x565200fa7ca8 in kvm_cpu_exec (cpu=cpu@entry=0x565202354480) at /root/qemu_debug_LSEEK/qemu_debug/qemu/accel/kvm/kvm-all.c:2089 #5 0x565200f7d1ce in qemu_kvm_cpu_thread_fn (arg=arg@entry=0x565202354480) at /root/qemu_debug_LSEEK/qemu_debug/qemu/cpus.c:1281 #6 0x565201398116 in qemu_thread_start (args=) at /root/qemu_debug_LSEEK/qemu_debug/qemu/util/qemu-thread-posix.c:502 #7 0x7f8a5e24a6ba in start_thread (arg=0x7f8a0700) at pthread_create.c:333 #8 0x7f8a5df8041d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 Thread 18 (Thread
Re: [raw] Guest stuck during live live-migration
[ Cc: qemu-block ] Am 23.11.2020 um 10:36 hat Quentin Grolleau geschrieben: > Hello, > > In our company, we are hosting a large number of Vm, hosted behind Openstack > (so libvirt/qemu). > A large majority of our Vms are runnign with local data only, stored on NVME, > and most of them are RAW disks. > > With Qemu 4.0 (can be even with older version) we see strange live-migration > comportement: First of all, 4.0 is relatively old. Generally it is worth retrying with the most recent code (git master or 5.2.0-rc2) before having a closer look at problems, because it is frustrating to spend considerable time debugging an issue and then find out it has already been fixed a year ago. > - some Vms live migrate at very high speed without issue (> 6 Gbps) > - some Vms are running correctly, but migrating at a strange low speed > (3Gbps) > - some Vms are migrating at a very low speed (1Gbps, sometime less) and > during the migration the guest is completely I/O stuck > > When this issue happen the VM is completly block, iostat in the Vm show us a > latency of 30 secs Can you get the stack backtraces of all QEMU threads while the VM is blocked (e.g. with gdb or pstack)? > First we thought it was related to an hardware issue we check it, we > comparing different hardware, but no issue where found there > > So one of my colleague had the idea to limit with "tc" the bandwidth on the > interface the migration was done, and it worked the VM didn't lose any ping > nor being I/O stuck > Important point : Once the Vm have been migrate (with the limitation ) one > time, if we migrate it again right after, the migration will be done at full > speed (8-9Gb/s) without freezing the Vm Since you say you're using local storage, I assume that you're doing both a VM live migration and storage migration at the same time. These are separate connections, storage is migrated using a NBD connection. Did you limit the bandwith for both connections, or if it was just one of them, which one was it? > It only happen on existing VM, we tried to replicate with a fresh instance > with exactly the same spec and nothing was happening > > We tried to replicate the workload inside the VM but there was no way to > replicate the case. So it was not related to the workload nor to the server > that hosts the Vm > > So we thought about the disk of the instance : the raw file. > > We also tried to strace -c the process during the live-migration and it was > doing a lot of "lseek" > > and we found this : > https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00462.html This case is different in that it used qcow2 (which should behave much better today). It also used ZFS, which you didn't mention. Is the problematic image stored on ZFS? If not, which filesystem is it? > So i rebuilt Qemu with this patch and the live-migration went well, at high > speed and with no VM freeze > ( https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ) > > Do you have a way to avoid the "lseek" mechanism as it consumes more > resources to find the holes in the disk and don't let any for the VM ? If you can provide the stack trace during the hang, we might be able to tell why we're even trying to find holes. Please also provide your QEMU command line. At the moment, my assumption is that this is during a mirror block job which is migrating the disk to your destination server. Not looking for holes would mean that a sparse source file would become fully allocated on the destination, which is usually not wanted (also we would potentially transfer a lot more data over the networkj). Can you give us a snippet from your strace that shows the individual lseek syscalls? Depending on which ranges are queried, maybe we could optimise things by caching the previous result. Also, a final remark, I know of some cases (on XFS) where lseeks were slow because the image file was heavily fragmented. Defragmenting the file resolved the problem, so this may be another thing to try. On XFS, newer QEMU versions set an extent size hint on newly created image files (during qemu-img create), which can reduce fragmentation considerably. Kevin > Server hosting the VM : > - Bi-Xeon hosts With NVME storage and 10 Go Network card > - Qemu 4.0 And Libvirt 5.4 > - Kernel 4.18.0.25 > > Guest having the issue : > - raw image with Debian 8 > > Here the qemu img on the disk : > > qemu-img info disk > image: disk > file format: raw > virtual size: 400G (429496729600 bytes) > disk size: 400G > > > Quentin GROLLEAU >
[raw] Guest stuck during live live-migration
Hello, In our company, we are hosting a large number of Vm, hosted behind Openstack (so libvirt/qemu). A large majority of our Vms are runnign with local data only, stored on NVME, and most of them are RAW disks. With Qemu 4.0 (can be even with older version) we see strange live-migration comportement: - some Vms live migrate at very high speed without issue (> 6 Gbps) - some Vms are running correctly, but migrating at a strange low speed (3Gbps) - some Vms are migrating at a very low speed (1Gbps, sometime less) and during the migration the guest is completely I/O stuck When this issue happen the VM is completly block, iostat in the Vm show us a latency of 30 secs First we thought it was related to an hardware issue we check it, we comparing different hardware, but no issue where found there So one of my colleague had the idea to limit with "tc" the bandwidth on the interface the migration was done, and it worked the VM didn't lose any ping nor being I/O stuck Important point : Once the Vm have been migrate (with the limitation ) one time, if we migrate it again right after, the migration will be done at full speed (8-9Gb/s) without freezing the Vm It only happen on existing VM, we tried to replicate with a fresh instance with exactly the same spec and nothing was happening We tried to replicate the workload inside the VM but there was no way to replicate the case. So it was not related to the workload nor to the server that hosts the Vm So we thought about the disk of the instance : the raw file. We also tried to strace -c the process during the live-migration and it was doing a lot of "lseek" and we found this : https://lists.gnu.org/archive/html/qemu-devel/2017-02/msg00462.html So i rebuilt Qemu with this patch and the live-migration went well, at high speed and with no VM freeze ( https://github.com/qemu/qemu/blob/master/block/file-posix.c#L2601 ) Do you have a way to avoid the "lseek" mechanism as it consumes more resources to find the holes in the disk and don't let any for the VM ? Server hosting the VM : - Bi-Xeon hosts With NVME storage and 10 Go Network card - Qemu 4.0 And Libvirt 5.4 - Kernel 4.18.0.25 Guest having the issue : - raw image with Debian 8 Here the qemu img on the disk : > qemu-img info disk image: disk file format: raw virtual size: 400G (429496729600 bytes) disk size: 400G Quentin GROLLEAU
