Re: Issues with KVM
On 22/07/2011 21:59, C Anthony Risinger wrote: is the disk's cache mode set to none [or maybe writeback] (virtmanager)? I tested various options: the most important is the cache mode. cache=none is a little bit better than cache=default (writethrough), but there is an huge difference using cache=unsafe. This option is not supported by virt-manager, but by kvm. Using cache=unsafe, the installation of FreeBSD is ~100x faster: - cache=default: write at 1 to 8 KB/sec - cache=none: write at 40 KB/sec - cache=unsafe: write at 1200 KB/sec According to agraf__ on IRC (#kvm on FreeNode), the cache mode has the following effect: - cache=writethrough calls fsync() after every write() - cache=none uses O_DIRECT - cache=writeback calls fsync() when the guest issues a barrier() (don't use O_DIRECT) - cache=unsafe doesn't do any fsync() (but don't use O_DIRECT) I still have to test writeback ;-) I found another mail thread, of last summer (July 2010), which is exactly the same problem that I had: BTRFS: Unbelievably slow with kvm/qemu on this mailing list. It contains another advice: Make sure you build your file system with mkfs.btrfs -m single -d single /dev/whatever. You may well be writing duplicate copies of everything. http://lkml.org/lkml/2010/7/12/5 -- FreeBSD installation in VirtualBox is as fast (or maybe a little bit slower) than the installation in kvm using cache=unsafe. I suppose that VirtualBox uses something like cache=unsafe or cache=writeback. Victor -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
On Mon, Jul 25, 2011 at 04:04:49PM +0200, Victor Stinner wrote: According to agraf__ on IRC (#kvm on FreeNode), the cache mode has the following effect: - cache=writethrough calls fsync() after every write() It uses O_DSYNC, which an be appromiately described as an fdatasync after every write. Not that it makes a different for most workloads on btrfs. - cache=none uses O_DIRECT - cache=writeback calls fsync() when the guest issues a barrier() (don't use O_DIRECT) Issue a cache flush. Barriers were an Linux-internal concept that is now gone. And there never was a barriere() call. FreeBSD installation in VirtualBox is as fast (or maybe a little bit slower) than the installation in kvm using cache=unsafe. I suppose that VirtualBox uses something like cache=unsafe or cache=writeback. What filesystem do you use in FreeBSD? FFS traditionally never issued cache flushes, so cache=writeback is equal to cache=unsafe for it. ZFS can issue cache flushes on Solaris, but I'm not sure if this was ported correctly to FreeBSD. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
Hi, Le 22/07/2011 21:59, C Anthony Risinger a écrit : ) is the host FS btrfs? The host (Fedora) FS is btrfs, the VM (Debian) uses ext4 (or maybe ext3, I will have to check). ) are virtio modules in the initramfs (or kernel probably)? ) are you sure virtio is being used (eg. are the disks called vdX vs sdX)? ) is the disk bus set to virtio (virtmanager)? I'm not sure. I will have to check. ) is the disk's cache mode set to none [or maybe writeback] (virtmanager)? I don't remember, I will check. What is the effect of this option? (What is the effect on the host FS (btrfs)?) ) is the disk's storage format set to raw, should be (virtmanager)? Yes, the image type is raw. ) is caching enabled on the image? () What do you mean? Is it possible to enable or disable the cache on a specific file on the host (Fedora, btrfs)? Or you mean set cache to none in the configuration of the VM? probably need to change the cache mode on the disk, or if the host is btrfs you need to flag the image with whetever is needed to prevent continuous COWing. Can nodatacow option of btrfs have an effect on this issue? Is it possible to set this option for a specific file or directory? Victor PS: I have subscribed to the mailing list. PS2: I'm not at home this week-end, I will check monday. -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Issues with KVM
Hi, I have a new fast computer to run many virtual machines. Everything looks very fast, except the installation of new operating systems in KVM. The installation is very fast until it begins to write on disk. It looks like it writes slower and slower. I tried Debian, FreeBSD, OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays the speed: it starts at 780 KB/sec (which it already very slow) to finish between 1 and 8 KB/sec. darksatanic suggested me to use nodatacow mount flag: it is not faster, and it looks even slower (fewer wsec/s in iostat output, see below). The computer is an Intel i7 2600 (4 cores with hyper threading: 8 threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are connected to SATA 6 GB/sec connectors using a P67 chipset. I'm using RAID 0 with Linux software (MD) RAID, and I have one unique btrfs partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38). I'm using hardware virtualisation with KVM. Debian is installed using virtio, so it should not be an issue with the hard drive driver of KVM. I'm watching iostat during the Debian installation. With the default mount option, wsec/s starts at 49000 to finish near 42000 (on the MD device), %usage is greater than 50% of both disks (near 80% for sda, near 60% for sdb). Using nodatacow option, wsec/s starts at 12000 (%usage 75%) to finish near 1 (%usage always 75%). It is slower, right? A sector is 512 bytes. The Debian image size is 40 GB, its type is raw. The system load is greater than 2 (or maybe 3) during the installation of the VM, while CPU usage is under 8% and wa% is also low (maybe 10% or lower, I don't remember). bonnie++ output (on the Fedora host, not in a VM): Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ned 24048M 346 98 220839 24 98489 19 245 84 251547 18 199.2 259 Latency 37256us 326ms 943ms 251ms 197ms 151ms Version 1.96 --Sequential Create-- Random Create ned -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 11128 34 + +++ 16558 45 14006 40 + +++ 17226 48 Latency 14997us 663us 11401us8115us 282us 10105us 1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,11128,34,+,+++,16558,45,14006,40,+,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us Do you have any idea why the %usage is so high (in iostat), while the speed looks so low? The disk speed during the installation is between 5 MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec according to bonnie++ on sequential output). It's difficult to read bonnie++ output: random create is near 14 MB/sec if I read correctly. btrfs behaves maybe very badly with a raw image of 40 GB, especially on RAID 0 with 2 disks? Should I try other KVM option (e.g. use another type of image)? Try btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or maybe not using btrfs for KVM images? :-) Victor -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner victor.stin...@haypocalc.com wrote: Hi, I have a new fast computer to run many virtual machines. Everything looks very fast, except the installation of new operating systems in KVM. The installation is very fast until it begins to write on disk. It looks like it writes slower and slower. I tried Debian, FreeBSD, OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays the speed: it starts at 780 KB/sec (which it already very slow) to finish between 1 and 8 KB/sec. ) is the host FS btrfs? ) are virtio modules in the initramfs (or kernel probably)? ) are you sure virtio is being used (eg. are the disks called vdX vs sdX)? ) is the disk bus set to virtio (virtmanager)? ) is the disk's cache mode set to none [or maybe writeback] (virtmanager)? ) is the disk's storage format set to raw, should be (virtmanager)? ) is caching enabled on the image? () probably need to change the cache mode on the disk, or if the host is btrfs you need to flag the image with whetever is needed to prevent continuous COWing. C Anthony -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
On Fri, Jul 22, 2011 at 2:59 PM, C Anthony Risinger anth...@xtfx.me wrote: On Fri, Jul 22, 2011 at 2:44 PM, Victor Stinner victor.stin...@haypocalc.com wrote: ) is caching enabled on the image? () oops, disregard that ... remainder left over from editing copy/paste :-) C Anthony -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
On 07/22/2011 03:44 PM, Victor Stinner wrote: Hi, I have a new fast computer to run many virtual machines. Everything looks very fast, except the installation of new operating systems in KVM. The installation is very fast until it begins to write on disk. It looks like it writes slower and slower. I tried Debian, FreeBSD, OpenIndiana and OpenBSD: same problem. The FreeBSD installer displays the speed: it starts at 780 KB/sec (which it already very slow) to finish between 1 and 8 KB/sec. darksatanic suggested me to use nodatacow mount flag: it is not faster, and it looks even slower (fewer wsec/s in iostat output, see below). The computer is an Intel i7 2600 (4 cores with hyper threading: 8 threads), 12 GB or RAM, 2 hard drives of 1 TB (Western Digital Caviar Blue 1 To 7200 RPM 32 Mo Serial ATA 6Gb/s - WD10EALX). Both disks are connected to SATA 6 GB/sec connectors using a P67 chipset. I'm using RAID 0 with Linux software (MD) RAID, and I have one unique btrfs partition of 2 TB. The host OS is Fedora 15 (Linux kernel 2.6.38). I'm using hardware virtualisation with KVM. Debian is installed using virtio, so it should not be an issue with the hard drive driver of KVM. I'm watching iostat during the Debian installation. With the default mount option, wsec/s starts at 49000 to finish near 42000 (on the MD device), %usage is greater than 50% of both disks (near 80% for sda, near 60% for sdb). Using nodatacow option, wsec/s starts at 12000 (%usage 75%) to finish near 1 (%usage always 75%). It is slower, right? A sector is 512 bytes. The Debian image size is 40 GB, its type is raw. The system load is greater than 2 (or maybe 3) during the installation of the VM, while CPU usage is under 8% and wa% is also low (maybe 10% or lower, I don't remember). bonnie++ output (on the Fedora host, not in a VM): Version 1.96 --Sequential Output-- --Sequential Input- --Random- Concurrency 1 -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks-- MachineSize K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP ned 24048M 346 98 220839 24 98489 19 245 84 251547 18 199.2 259 Latency 37256us 326ms 943ms 251ms 197ms 151ms Version 1.96 --Sequential Create-- Random Create ned -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete-- files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP 16 11128 34 + +++ 16558 45 14006 40 + +++ 17226 48 Latency 14997us 663us 11401us8115us 282us 10105us 1.96,1.96,ned,1,1311365747,24048M,,346,98,220839,24,98489,19,245,84,251547,18,199.2,259,16,11128,34,+,+++,16558,45,14006,40,+,+++,17226,48,37256us,326ms,943ms,251ms,197ms,151ms,14997us,663us,11401us,8115us,282us,10105us Do you have any idea why the %usage is so high (in iostat), while the speed looks so low? The disk speed during the installation is between 5 MB/sec and 23 MB/sec, whereas the raw speed is greater than 200 MB/sec on the host (234 MB/sec according to hdparm -t /dev/md127, 220 MB/sec according to bonnie++ on sequential output). It's difficult to read bonnie++ output: random create is near 14 MB/sec if I read correctly. btrfs behaves maybe very badly with a raw image of 40 GB, especially on RAID 0 with 2 disks? Should I try other KVM option (e.g. use another type of image)? Try btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or maybe not using btrfs for KVM images? :-) Use the kvm option of cache=none for your device. Granted its still going to be slow, but it should be a bit faster. Thanks, Josef -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: Issues with KVM
On Fri, 22 Jul 2011 21:44:24 +0200, Victor Stinner wrote: Should I try other KVM option (e.g. use another type of image)? Try btrfs RAID instead of Linux MD RAID? Try to disable some CPU cores? Or maybe not using btrfs for KVM images? :-) Hi, I would suggest you the following points: - qemu-img create -f qcow2 -o size=400,preallocation=metadata vdisk.img - disk: cache=none - controller: virtio Best regards, Morten -- To unsubscribe from this list: send the line unsubscribe linux-btrfs in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
