Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
On 05/28/2018 07:05 AM, Kashyap Chamarthy wrote: Cc the QEMU Block Layer mailing list (qemu-bl...@nongnu.org), who might have more insights here; and wrap long lines. ... 170 to less than 10. I also show the figure of this experiment in the attachment of this email. [The attachment should arrive on the 'libvirt-users' list archives; but it's not there yet -- https://www.redhat.com/archives/libvirt-users/2018-May/thread.html] Actually, the attachment was probably rejected by list moderation for being oversized. -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] Using libvirt to access Ceph RBDs with Xen
On 05/27/2018 05:42 AM, thg wrote: Hi everybody, my background: I'm doing Xen since 10++ years, many years with DRBD for high availability, since some time I'm using preferable GlusterFS with FUSE as replicated storage, where I place the image-files for the vms. In my current project we started (successfully) with Xen/GlusterFS too, but because the provider, where we placed the servers, uses widely CEPH, we decided to switch too. Unfortunately I'm new to CEPH, but with help of a technician, we have running a 3 node CEPH-cluster now, that seems to work fine. Hardware: - Xeons, 24 Cores, 256 GB RAM, 2x 240 GB system-SSDs RAID1, 4x 1,92 TB data-SSDs (no RAID) Software we are using: - CentOS 7.5.1804 - Kernel: 4.9.86-30.el7 @centos-virt-xen-48 - Xen: 4.8.3-5.el7 @centos-virt-xen-48 - libvirt-xen: 4.1.0-2.xen48.el7@centos-virt-xen-48 - Ceph: 2:12.2.5-0.el7 @Ceph CEPH itself works, Xen too and even libvirt (I'm quite new in this too, normally I use the XL-tools to manage my VMs) ;-) I can manually map an RBD and start a PV-VM via libvirt, after modifying the XML-config and defined the VM with libvirt. That is what happens: # virsh define xml/testvm3.xml Domain testvm3 defined from xml/testvm3.xml # virsh start --console testvm3 error: Failed to start domain testvm3 error: internal error: libxenlight failed to create new domain 'testvm3' xml/testvm3.xml: -> ... ... Logs: /var/log/libvirt/libxl/libxl-driver.log: -> ... 2018-05-27 11:24:36.666+: libxl: libxl_bootloader.c:634:bootloader_finished: bootloader failed - consult logfile /var/log/xen/bootloader.14.log 2018-05-27 11:24:36.666+: libxl: libxl_exec.c:118:libxl_report_child_exitstatus: bootloader [183216] exited with error status 1 2018-05-27 11:24:36.667+: libxl: libxl_create.c:1259:domcreate_rebuild_done: cannot (re-)build domain: -3 /var/log/xen/bootloader.14.log: -> Traceback (most recent call last): File "/usr/lib64/xen/bin/pygrub", line 896, in part_offs = get_partition_offsets(file) File "/usr/lib64/xen/bin/pygrub", line 113, in get_partition_offsets image_type = identify_disk_image(file) File "/usr/lib64/xen/bin/pygrub", line 56, in identify_disk_image fd = os.open(file, os.O_RDONLY) OSError: [Errno 2] No such file or directory: 'rbd:devel-pool/testvm3.rbd:id=libvirt:key=AQBThwFbGFRYFxxx==:auth_supported=cephx\\;none:mon_host=10.20.30.1\\:6789\\;10.20.30.2\\:6789\\;10.20.30.3\\:6789' pygrub doesn't work with rbd. But why not use pvgrub instead, which will work with rbd, and doesn't require mounting foreign (possibly hostile) images in dom0? Regards, Jim ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] [libvirt] virRandomBits - not very random
On 05/29/2018 09:44 AM, Michal Privoznik wrote: > On 05/29/2018 03:38 PM, Martin Kletzander wrote: >> On Fri, May 25, 2018 at 09:37:44AM -0500, Eric Blake wrote: >>> On 05/25/2018 09:17 AM, Michal Privoznik wrote: >>> >> We should probably seed it with data from /dev/urandom, and/or the new >> Linux getrandom() syscall (or BSD equivalent). I'm not quite sure that right after reboot there's going to be enough entropy. Every service that's starting wants some random bits. But it's probably better than what we have now. >>> >>> Here's where we left things last time it came up: >>> >>> https://www.redhat.com/archives/libvir-list/2014-December/msg00573.html >>> >>> If gnutls has an interface that will give us random bits >>> (gnutls_key_generate() in 3.0, perhaps), we should use THAT for all of >>> our random bits (and forget about a seed), except when we are mocking >>> things in our testsuite, and need a deterministic PRNG from a >>> deterministic seed. >>> >>> If not (including if we are not linked with gnutls), then we should >>> prefer the new Linux syscall but fall back to /dev/urandom for JUST >>> enough bits for a seed; once we're seeded, stick with using our existing >>> PRNG for all future bits (after all, we aren't trying to generate >>> cryptographically secure keys using virRandomBits - and the places where >>> we DO need crypto-strong randomness such as setting up TLS migration is >>> where we are relying on gnutls to provide it rather than virRandomBits). >>> >>> So at this point, it's just a matter of someone writing the patches. >>> >> >> Actually, do we need to have a fallback at all? Can't we just drop all the >> gross parts of the code the conditionally compile based on GNUTLS >> support? Why >> don't we have gnutls required? > > That's exactly what I'm suggesting in my patches [1]. gnutls is widely > available (including Linux, Windows, *BSD, Mac Os X). However, before > doing that we need to fix virRandomBits() to actually call gnutls_rnd(). > > 1: https://www.redhat.com/archives/libvir-list/2018-May/msg02077.html > I have this faint recollection of one of the CI platform builds failing because something in the gnutls* family didn't exist there when I was making the changes to add the domain master secret code After a bit of digging, it seems it was a perhaps a CENTOS6 environment: https://www.redhat.com/archives/libvir-list/2016-April/msg00287.html and since IIUC that's not an issue any more John now if I could only figure out why my mail client seems to be dropping any patches with "crypto" in the subject line (I'm missing patches 2-4 and 10 from the series referenced above)... ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] [libvirt] virRandomBits - not very random
On 05/29/2018 03:38 PM, Martin Kletzander wrote: > On Fri, May 25, 2018 at 09:37:44AM -0500, Eric Blake wrote: >> On 05/25/2018 09:17 AM, Michal Privoznik wrote: >> > We should probably seed it with data from /dev/urandom, and/or the new > Linux getrandom() syscall (or BSD equivalent). >>> >>> I'm not quite sure that right after reboot there's going to be enough >>> entropy. Every service that's starting wants some random bits. But it's >>> probably better than what we have now. >> >> Here's where we left things last time it came up: >> >> https://www.redhat.com/archives/libvir-list/2014-December/msg00573.html >> >> If gnutls has an interface that will give us random bits >> (gnutls_key_generate() in 3.0, perhaps), we should use THAT for all of >> our random bits (and forget about a seed), except when we are mocking >> things in our testsuite, and need a deterministic PRNG from a >> deterministic seed. >> >> If not (including if we are not linked with gnutls), then we should >> prefer the new Linux syscall but fall back to /dev/urandom for JUST >> enough bits for a seed; once we're seeded, stick with using our existing >> PRNG for all future bits (after all, we aren't trying to generate >> cryptographically secure keys using virRandomBits - and the places where >> we DO need crypto-strong randomness such as setting up TLS migration is >> where we are relying on gnutls to provide it rather than virRandomBits). >> >> So at this point, it's just a matter of someone writing the patches. >> > > Actually, do we need to have a fallback at all? Can't we just drop all the > gross parts of the code the conditionally compile based on GNUTLS > support? Why > don't we have gnutls required? That's exactly what I'm suggesting in my patches [1]. gnutls is widely available (including Linux, Windows, *BSD, Mac Os X). However, before doing that we need to fix virRandomBits() to actually call gnutls_rnd(). 1: https://www.redhat.com/archives/libvir-list/2018-May/msg02077.html Michal ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] [libvirt] virRandomBits - not very random
On Fri, May 25, 2018 at 09:37:44AM -0500, Eric Blake wrote: On 05/25/2018 09:17 AM, Michal Privoznik wrote: We should probably seed it with data from /dev/urandom, and/or the new Linux getrandom() syscall (or BSD equivalent). I'm not quite sure that right after reboot there's going to be enough entropy. Every service that's starting wants some random bits. But it's probably better than what we have now. Here's where we left things last time it came up: https://www.redhat.com/archives/libvir-list/2014-December/msg00573.html If gnutls has an interface that will give us random bits (gnutls_key_generate() in 3.0, perhaps), we should use THAT for all of our random bits (and forget about a seed), except when we are mocking things in our testsuite, and need a deterministic PRNG from a deterministic seed. If not (including if we are not linked with gnutls), then we should prefer the new Linux syscall but fall back to /dev/urandom for JUST enough bits for a seed; once we're seeded, stick with using our existing PRNG for all future bits (after all, we aren't trying to generate cryptographically secure keys using virRandomBits - and the places where we DO need crypto-strong randomness such as setting up TLS migration is where we are relying on gnutls to provide it rather than virRandomBits). So at this point, it's just a matter of someone writing the patches. Actually, do we need to have a fallback at all? Can't we just drop all the gross parts of the code the conditionally compile based on GNUTLS support? Why don't we have gnutls required? -- Eric Blake, Principal Software Engineer Red Hat, Inc. +1-919-301-3266 Virtualization: qemu.org | libvirt.org -- libvir-list mailing list libvir-l...@redhat.com https://www.redhat.com/mailman/listinfo/libvir-list signature.asc Description: Digital signature ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
> -Original Messages- > From: "Kashyap Chamarthy" > Sent Time: 2018-05-28 21:19:14 (Monday) > To: "Chunguang Li" > Cc: libvirt-users@redhat.com, qemu-bl...@nongnu.org, dgilb...@redhat.com > Subject: Re: [libvirt-users] VM I/O performance drops dramatically during > storage migration with drive-mirror > > On Mon, May 28, 2018 at 02:05:05PM +0200, Kashyap Chamarthy wrote: > > Cc the QEMU Block Layer mailing list (qemu-bl...@nongnu.org), > > [Sigh; now add the QEMU BLock Layer e-mail list to Cc, without typos.] Yes, thank you very much. > > > who might > > have more insights here; and wrap long lines. > > > > On Mon, May 28, 2018 at 06:07:51PM +0800, Chunguang Li wrote: > > > Hi, everyone. > > > > > > Recently I am doing some tests on the VM storage+memory migration with > > > KVM/QEMU/libvirt. I use the following migrate command through virsh: > > > "virsh migrate --live --copy-storage-all --verbose vm1 > > > qemu+ssh://192.168.1.91/system tcp://192.168.1.91". I have checked the > > > libvirt debug output, and make sure that the drive-mirror + NBD > > > migration method is used. > > > > > > Inside the VM, I use an I/O benchmark (Iometer) to generate an oltp > > > workload. I record the I/O performance (IOPS) before/during/after > > > migration. When the migration begins, the IOPS dropped by 30%-40%. > > > This is reasonable, because the migration I/O competes with the > > > workload I/O. However, during almost the last period of migration > > > (which is 66s in my case), the IOPS dropped dramatically, from about > > > 170 to less than 10. I also show the figure of this experiment in the > > > attachment of this email. > > > > [The attachment should arrive on the 'libvirt-users' list archives; but > > it's not there yet -- > > https://www.redhat.com/archives/libvirt-users/2018-May/thread.html] The figure of the experiment is also available at: https://pan.baidu.com/s/1pByKQtJ7VdFCDbX-ZMyOwQ > > > > > I want to figure out what results in this period with very low IOPS. > > > First, I added some printf()s in the QEMU code, and knew that, this > > > period occurs just before the memory migration phase. (BTW, the memory > > > migration is very fast, which is just about 5s.) So I think this > > > period should be the last phase of the "drive-mirror" process of QEMU. > > > So then I tried to read the code of "drive-mirror" in QEMU, but failed > > > to understand it very well. > > > > > > Does anybody know what may lead to this period with very low IOPS? > > > Thank you very much. > > > > > > Some details of this experiment: The VM disk image file is 30GB > > > (format = raw,cache=none,aio=native), and Iometer operates on an 10GB > > > file inside the VM. The oltp workload consists of 33% writes and 67% > > > reads (8KB request size, all random). The VM memory size is 4GB, most > > > of which should be zero pages, so the memory migration is very fast. > > > > > > -- > > > Chunguang Li, Ph.D. Candidate > > > Wuhan National Laboratory for Optoelectronics (WNLO) > > > Huazhong University of Science & Technology (HUST) > > > Wuhan, Hubei Prov., China > > > > > > -- > > /kashyap > > -- > /kashyap ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
Re: [libvirt-users] VM I/O performance drops dramatically during storage migration with drive-mirror
> -Original Messages- > From: "Kashyap Chamarthy" > Sent Time: 2018-05-28 20:05:05 (Monday) > To: "Chunguang Li" > Cc: libvirt-users@redhat.com, qemu-bl...@redhat.com, dgilb...@redhat.com > Subject: Re: [libvirt-users] VM I/O performance drops dramatically during > storage migration with drive-mirror > > Cc the QEMU Block Layer mailing list (qemu-bl...@nongnu.org), who might > have more insights here; and wrap long lines. > Hi, Kashyap, thank you very much. > On Mon, May 28, 2018 at 06:07:51PM +0800, Chunguang Li wrote: > > Hi, everyone. > > > > Recently I am doing some tests on the VM storage+memory migration with > > KVM/QEMU/libvirt. I use the following migrate command through virsh: > > "virsh migrate --live --copy-storage-all --verbose vm1 > > qemu+ssh://192.168.1.91/system tcp://192.168.1.91". I have checked the > > libvirt debug output, and make sure that the drive-mirror + NBD > > migration method is used. > > > > Inside the VM, I use an I/O benchmark (Iometer) to generate an oltp > > workload. I record the I/O performance (IOPS) before/during/after > > migration. When the migration begins, the IOPS dropped by 30%-40%. > > This is reasonable, because the migration I/O competes with the > > workload I/O. However, during almost the last period of migration > > (which is 66s in my case), the IOPS dropped dramatically, from about > > 170 to less than 10. I also show the figure of this experiment in the > > attachment of this email. > > [The attachment should arrive on the 'libvirt-users' list archives; but > it's not there yet -- > https://www.redhat.com/archives/libvirt-users/2018-May/thread.html] > I don't know whether the attachment could arrive on the list archives. So I make it available through this link: https://pan.baidu.com/s/1pByKQtJ7VdFCDbX-ZMyOwQ Chunguang > > I want to figure out what results in this period with very low IOPS. > > First, I added some printf()s in the QEMU code, and knew that, this > > period occurs just before the memory migration phase. (BTW, the memory > > migration is very fast, which is just about 5s.) So I think this > > period should be the last phase of the "drive-mirror" process of QEMU. > > So then I tried to read the code of "drive-mirror" in QEMU, but failed > > to understand it very well. > > > > Does anybody know what may lead to this period with very low IOPS? > > Thank you very much. > > > > Some details of this experiment: The VM disk image file is 30GB > > (format = raw,cache=none,aio=native), and Iometer operates on an 10GB > > file inside the VM. The oltp workload consists of 33% writes and 67% > > reads (8KB request size, all random). The VM memory size is 4GB, most > > of which should be zero pages, so the memory migration is very fast. > > > > -- > > Chunguang Li, Ph.D. Candidate > > Wuhan National Laboratory for Optoelectronics (WNLO) > > Huazhong University of Science & Technology (HUST) > > Wuhan, Hubei Prov., China > > > -- > /kashyap ___ libvirt-users mailing list libvirt-users@redhat.com https://www.redhat.com/mailman/listinfo/libvirt-users
