On Fri, Jun 08, 2012 at 09:30:19PM +0100, Brian Candler wrote: > ubuntu@lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros2 bs=1024k count=100 > 100+0 records in > 100+0 records out > 104857600 bytes (105 MB) copied, 14.5182 s, 7.2 MB/s > > And this is after live-migrating the VM to dev-storage2: > > ubuntu@lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros3 bs=1024k count=100 > 100+0 records in > 100+0 records out > 104857600 bytes (105 MB) copied, 4.17285 s, 25.1 MB/s
I did some more timings after converting the qcow2 image to a raw file. Note that you have to be careful: qemu-img convert -O raw will give you a sparse file, not actually allocating space on disk. So I had to flatten it with dd (which incidentally showed a reasonable write throughput of ~350MB/sec to the 12-disk RAID10 array, and was the same writing locally or writing to a single-brick gluster volume) Tests: 1. VM using a single-brick gluster volume as backend. The brick is on the same node as KVM is running. (Actually the second cluster node was powered off for all these tests) ubuntu@lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros4 bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 55.9581 s, 9.4 MB/s (Strangely this is lower than the 25MB/s I got before) 2. VM image stored directly on the RAID10 array - no gluster. ubuntu@lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros4 bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 10.6027 s, 49.4 MB/s 3. Same VM instance after test 2, but this time with option cache='none' (which doesn't work with glusterfs) ubuntu@lucidtest:~$ dd if=/dev/zero of=/var/tmp/test.zeros5 bs=1024k count=500 500+0 records in 500+0 records out 524288000 bytes (524 MB) copied, 2.29959 s, 228 MB/s That's more like it :-) So clearly cache='none' (O_DIRECT) makes a big difference when using a local filesystem, so I'd very much like to be able to test it with gluster. I'd also very much look forward to having libglusterfs integrated directly into KVM, which I believe is on the cards at some point: http://www.mail-archive.com/[email protected]/msg01812.html Regards, Brian. P.S. for those who haven't seen it yet, there's a very nice Red Hat presentation on KVM performance tuning here. http://www.linux-kvm.org/wiki/images/5/59/Kvm-forum-2011-performance-improvements-optimizations-D.pdf _______________________________________________ Gluster-users mailing list [email protected] http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
