Re: [Openstack] KVM disk performance
Have you tried virtio for storage? http://www.linux-kvm.org/page/Virtio Enviado desde mi iPhone, perdona la brevedad El 26/03/2012, a las 14:59, Martin van Wilderen - JDN BV mar...@jdn.nl escribió: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Hi Diego, Thanks for your quick response. Yes we are using the virtio drivers. We have to add them manual for FreeBSD Windows, and it works. Only the performance is very bad. Kind regards, Martin On Mar 26, 2012, at 3:43 PM, Diego Parrilla wrote: Have you tried virtio for storage? http://www.linux-kvm.org/page/Virtio Enviado desde mi iPhone, perdona la brevedad El 26/03/2012, a las 14:59, Martin van Wilderen - JDN BV mar...@jdn.nl escribió: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
On 03/26/2012 01:59 PM, Martin van Wilderen - JDN BV wrote:
Hi List,
I have a question about KVM disk performance. We are using Openstack Nova on
three machines. These machines have a SSD drive with a dd write performance
of about 130mb/s
Within the instance the write performance is down to about 5 mb/s. When using
the allocate trick (dd zero to disk before newfs) we get a performance of 20
mb/s.
Interesting. I presume this performance difference is when growing the
allocation.
I.E. without the dd allocate trick, writes to new files = 5MB/s, but old
files = 20MB/s ?
What file did you do the `dd` on exactly and what size was it?
I could do a patch to nova to do something like:
fallocate --version || exit 0
fallocate -l $SIZE $DISKIMG || { rm -f $DISKIMG; exit 1; }
But only when I get a better understand of your setup.
Note it would need to be optional too, akin to a memory
overcommit like flag.
Note the above change would also give the benefit of
immediate feedback of ENOSPC
Things a have tried but don't give any extra results are:
- Settings the disklayout from qcow2 to raw
I'm a bit surprised that had no effect.
- Settings the cache type in libvirt.xml (writeback, writethrough, none)
- Switching KSM on and off.
- Tested with different guests OS, Linux, FreeBSD, Windows.
Is there someone who had some extra info i can check? Or are there more
people with this issue?
Snippet from libvirt.xml
driver type='qcow2'/ cache='writeback'/
source file='/var/lib/nova/instances/instance-0113/disk'/
target dev='vda' bus='virtio'/
I'll let others more knowledgeable comment on
general virt IO overheads.
cheers,
Pádraig.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Hi Martin 1. Does /var/lib/nova/instances reside on your SSD? (just double checking it's not instead pointing to a normal stoage device). 2. Do you see the expected performance on the host operating system? 3. Could you please provide the complete libvirt .xml file? 4. Could you please provide the operating system and version of the guest? Cheers David On Mon, Mar 26, 2012 at 1:59 PM, Martin van Wilderen - JDN BV mar...@jdn.nl wrote: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
On 03/26/2012 01:59 PM, Martin van Wilderen - JDN BV wrote:
Hi List,
I have a question about KVM disk performance. We are using Openstack Nova on
three machines. These machines have a SSD drive with a dd write performance
of about 130mb/s
Within the instance the write performance is down to about 5 mb/s. When using
the allocate trick (dd zero to disk before newfs) we get a performance of 20
mb/s.
Interesting. I presume this performance difference is when growing the
allocation.
I.E. without the dd allocate trick, writes to new files = 5MB/s, but old
files = 20MB/s ?
What file did you do the `dd` on exactly and what size was it?
I could do a patch to nova to do something like:
fallocate --version || exit 0
fallocate -l $SIZE $DISKIMG || { rm -f $DISKIMG; exit 1; }
But only when I get a better understand of your setup.
Note it would need to be optional too, akin to a memory
overcommit like flag.
Note the above change would also give the benefit of
immediate feedback of ENOSPC
Things a have tried but don't give any extra results are:
- Settings the disklayout from qcow2 to raw
I'm a bit surprised that had no effect.
- Settings the cache type in libvirt.xml (writeback, writethrough, none)
- Switching KSM on and off.
- Tested with different guests OS, Linux, FreeBSD, Windows.
Is there someone who had some extra info i can check? Or are there more
people with this issue?
Snippet from libvirt.xml
driver type='qcow2'/ cache='writeback'/
source file='/var/lib/nova/instances/instance-0113/disk'/
target dev='vda' bus='virtio'/
I'll let others more knowledgeable comment on
general virt IO overheads.
cheers,
Pádraig.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Can you provide information about what version of kvm you're using ? I suspect this is really unrelated to openstack specifically. On Mon, 26 Mar 2012, Martin van Wilderen - JDN BV wrote: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Hi David, 1. Yes. That's the only disk in the system. 2. Yes: nova1:/var/lib/nova/instances$ dd if=/dev/zero of=test bs=1M count=256 256+0 records in 256+0 records out 268435456 bytes (268 MB) copied, 2.04146 s, 131 MB/s 4. I have tried with: - Linux Ubuntu 10.10 - FreeBSD 8.2 with virtio driver - Windows 2008 R2 Server with virtio driver 3. my libvirt.xml domain type='kvm' nameinstance-0121/name memory1048576/memory os typehvm/type boot dev=hd / /os features acpi/ /features vcpu2/vcpu devices disk type='file' driver type='raw'/ source file='/var/lib/nova/instances/instance-0121/disk'/ target dev='vda' bus='virtio'/ /disk disk type='file' driver type='raw'/ source file='/var/lib/nova/instances/instance-0121/disk.local'/ target dev='vdb' bus='virtio'/ /disk interface type='bridge' source bridge='br100'/ mac address='02:16:3e:33:42:e3'/ filterref filter=nova-instance-instance-0121-02163e3342e3 parameter name=IP value=10.0.0.34 / parameter name=DHCPSERVER value=10.0.0.9 / /filterref /interface !-- The order is significant here. File must be defined first -- serial type=file source path='/var/lib/nova/instances/instance-0121/console.log'/ target port='1'/ /serial console type='pty' tty='/dev/pts/2' source path='/dev/pts/2'/ target port='0'/ /console serial type='pty' source path='/dev/pts/2'/ target port='0'/ /serial graphics type='vnc' port='-1' autoport='yes' keymap='en-us' listen='0.0.0.0'/ /devices /domain Met vriendelijke groet, Martin van Wilderen Technical Manager On Mar 26, 2012, at 4:45 PM, David Busby wrote: Hi Martin Does /var/lib/nova/instances reside on your SSD? (just double checking it's not instead pointing to a normal stoage device). Do you see the expected performance on the host operating system? Could you please provide the complete libvirt .xml file? Could you please provide the operating system and version of the guest? Cheers David On Mon, Mar 26, 2012 at 1:59 PM, Martin van Wilderen - JDN BV mar...@jdn.nl wrote: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Hi Scott, kvm --version QEMU emulator version 0.13.91 (qemu-kvm-0.14.0-rc1), Copyright (c) 2003-2008 Fabrice Bellard virsh --version 0.8.8 uname -r 2.6.35-22-server Need any more info? Met vriendelijke groet, Martin van Wilderen Technical Manager On Mar 26, 2012, at 5:12 PM, Scott Moser wrote: Can you provide information about what version of kvm you're using ? I suspect this is really unrelated to openstack specifically. On Mon, 26 Mar 2012, Martin van Wilderen - JDN BV wrote: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
Hi Martin:
Which os version on your host machines? I tested kvm on rhel 6.1 have
this issue, guest os is freebsd virtio, on rhel 6.2 performance is well.
2012/3/26 Pádraig Brady p...@draigbrady.com
On 03/26/2012 01:59 PM, Martin van Wilderen - JDN BV wrote:
Hi List,
I have a question about KVM disk performance. We are using Openstack
Nova on three machines. These machines have a SSD drive with a dd write
performance of about 130mb/s
Within the instance the write performance is down to about 5 mb/s. When
using the allocate trick (dd zero to disk before newfs) we get a
performance of 20 mb/s.
Interesting. I presume this performance difference is when growing the
allocation.
I.E. without the dd allocate trick, writes to new files = 5MB/s, but old
files = 20MB/s ?
What file did you do the `dd` on exactly and what size was it?
I could do a patch to nova to do something like:
fallocate --version || exit 0
fallocate -l $SIZE $DISKIMG || { rm -f $DISKIMG; exit 1; }
But only when I get a better understand of your setup.
Note it would need to be optional too, akin to a memory
overcommit like flag.
Note the above change would also give the benefit of
immediate feedback of ENOSPC
Things a have tried but don't give any extra results are:
- Settings the disklayout from qcow2 to raw
I'm a bit surprised that had no effect.
- Settings the cache type in libvirt.xml (writeback, writethrough, none)
- Switching KSM on and off.
- Tested with different guests OS, Linux, FreeBSD, Windows.
Is there someone who had some extra info i can check? Or are there more
people with this issue?
Snippet from libvirt.xml
driver type='qcow2'/ cache='writeback'/
source file='/var/lib/nova/instances/instance-0113/disk'/
target dev='vda' bus='virtio'/
I'll let others more knowledgeable comment on
general virt IO overheads.
cheers,
Pádraig.
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
___
Mailing list: https://launchpad.net/~openstack
Post to : openstack@lists.launchpad.net
Unsubscribe : https://launchpad.net/~openstack
More help : https://help.launchpad.net/ListHelp
Re: [Openstack] KVM disk performance
On Mon, 26 Mar 2012, Martin van Wilderen - JDN BV wrote: Hi Scott, kvm --version QEMU emulator version 0.13.91 (qemu-kvm-0.14.0-rc1), Copyright (c) 2003-2008 Fabrice Bellard virsh --version 0.8.8 uname -r 2.6.35-22-server Need any more info? Well, unfortunately there is lots more info needed, and I'm not really able to dedicate resources to debugging this at this point. I was curious because I had circumstantial evidence pointing to qemu-kvm 1.0 (in ubuntu 12.04) being slower than 11.10. However, your data above is similar to what I've seen in 12.04. Most definitely, if you want write performance, you will need fully allocated disks. The qcow sparse is good for quick deployment, but not good for write speed. You may also want to try using qed format disk as that is supposed to be a bit better. I had one kvm person suggest looking at barriers of the host's ext4 filesystem, saying: if you have ext4 filesystem underneath with barriers enabled, then you'll find that the guest will submit barriers to the host filesystem which will end up flushing the device cache A quick google showed http://blog.smartlogicsolutions.com/2009/06/04/mount-options-to-improve-ext4-file-system-performance on generally how to get some additional performance out of ext4. I'm interested in hearing what you find out, but unfortunately at the moment, I dont' have a magic bullet for you. Met vriendelijke groet, Martin van Wilderen Technical Manager On Mar 26, 2012, at 5:12 PM, Scott Moser wrote: Can you provide information about what version of kvm you're using ? I suspect this is really unrelated to openstack specifically. On Mon, 26 Mar 2012, Martin van Wilderen - JDN BV wrote: Hi List, I have a question about KVM disk performance. We are using Openstack Nova on three machines. These machines have a SSD drive with a dd write performance of about 130mb/s Within the instance the write performance is down to about 5 mb/s. When using the allocate trick (dd zero to disk before newfs) we get a performance of 20 mb/s. Things a have tried but don't give any extra results are: - Settings the disklayout from qcow2 to raw - Settings the cache type in libvirt.xml (writeback, writethrough, none) - Switching KSM on and off. - Tested with different guests OS, Linux, FreeBSD, Windows. Is there someone who had some extra info i can check? Or are there more people with this issue? Snippet from libvirt.xml driver type='qcow2'/ cache='writeback'/ source file='/var/lib/nova/instances/instance-0113/disk'/ target dev='vda' bus='virtio'/ Kind regards, Martin ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp ___ Mailing list: https://launchpad.net/~openstack Post to : openstack@lists.launchpad.net Unsubscribe : https://launchpad.net/~openstack More help : https://help.launchpad.net/ListHelp
