Re: [ovirt-users] Bad performance with Windows 2012 guests
-device usb-tablet,id=input0 -vnc 172.16.1.14:2,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on Sven Thanks a lot. I will try trace this issue on my local setup. Best regards, Vadim. -Ursprüngliche Nachricht- Von: Vadim Rozenfeld [mailto:vroze...@redhat.com] Gesendet: Montag, 4. Mai 2015 05:00 An: Sven Achtelik Cc: Doron Fediuck; Martijn Grendelman; Karen Noel; users@ovirt.org Betreff: Re: AW: AW: Bad performance with Windows 2012 guests On Sun, 2015-05-03 at 07:46 -0500, Sven Achtelik wrote: Hi Vadim, I've tested the performance with CrystalDiskMark from inside the Windows guest. Using Win2k8 R2 I got expected values for my system, about 88 MB/s on 4k random with 32 queues and 500MB/s + sequential writes with 32 queues. Using a Windows 2012 VM on the same system it's only 33MB/s on 4k random with 32 queues and 300MB/s sequential writes. Similar tests with a linux VM show a bit better values than the Win2k8 R2 and respond ultra-fast. My hosts are connected via iSCSI using a 10 GbE link and a ZFS appliance as the storage system. All tests have been run several times with the same results. Sven, Can I ask you to post the Windows 2012 VM qemu command line? Thanks, Vadim. Sven -Ursprüngliche Nachricht- Von: Vadim Rozenfeld [mailto:vroze...@redhat.com] Gesendet: Sonntag, 3. Mai 2015 14:35 An: Sven Achtelik Cc: Doron Fediuck; Martijn Grendelman; Karen Noel; users@ovirt.org Betreff: Re: AW: Bad performance with Windows 2012 guests On Sun, 2015-05-03 at 06:48 -0500, Sven Achtelik wrote: Hi Doron, I've also noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm
Re: [ovirt-users] Bad performance with Windows 2012 guests
noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. [snip] Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Sure! Command line is this: /usr/libexec/qemu-kvm -name Getafix -S -M rhel6.5.0 -cpu Penryn,hv_relaxed -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 34951c25-9a37-4712-a16a-fdfc98f4febc -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-6.el6.centos.12.2,serial=44454C4C-3400-1058-804C-B1C04F42344A,uuid=34951c25-9a37-4712-a16a-fdfc98f4febc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Getafix.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-01-12T11:14:02,clock=vm,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/aefd5844-6e01-4070-b3b9-c0d73cc40c78/52678e67-a202-4306-b7ed-5fed8df10edf/images/28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5/4c7b571e-4b29-47b9-ab4b-5799d64f28f9,if=none,id=drive-virtio-disk0,format=raw,serial=28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:74:59:a2,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 172.17.6.14:7,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on Qemu version: qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 Please let me know if I can do more to help! Best regards, Martijn. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On 4 May 2015, at 09:35, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. [snip] Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Sure! Command line is this: /usr/libexec/qemu-kvm -name Getafix -S -M rhel6.5.0 -cpu Penryn,hv_relaxed -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 34951c25-9a37-4712-a16a-fdfc98f4febc -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-6.el6.centos.12.2,serial=44454C4C-3400-1058-804C-B1C04F42344A,uuid=34951c25-9a37-4712-a16a-fdfc98f4febc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Getafix.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-01-12T11:14:02,clock=vm,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/aefd5844-6e01-4070-b3b9-c0d73cc40c78/52678e67-a202-4306-b7ed- 5fed8df10edf/images/28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5/4c7b571e-4b29-47b9-ab4b-5799d64f28f9,if=none,id=drive-virtio-disk0,format=raw,serial=28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:74:59:a2,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet, id=input0 -vnc 172.17.6.14:7,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on Qemu version: qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 Please let me know if I can do more to help! How about trying virtio-scsi? Same difference? We'll be supporting virtio blk dataplane in 3.6, that may affect the performance significantly. Also a EL7 hypervisor could change results a lot. Do you have any at hand to give it a try? Well, also hyperv enlightnment? Not sure, but worth a try. It's currently disabled in osinfo entry for Win8/2012, can you try that?(on a non-production VM;) Thanks, michal Best regards, Martijn. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On Mon, 2015-05-04 at 09:34 +0200, Martijn Grendelman wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. [snip] Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Sure! Command line is this: /usr/libexec/qemu-kvm -name Getafix -S -M rhel6.5.0 -cpu Penryn,hv_relaxed -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 34951c25-9a37-4712-a16a-fdfc98f4febc -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-6.el6.centos.12.2,serial=44454C4C-3400-1058-804C-B1C04F42344A,uuid=34951c25-9a37-4712-a16a-fdfc98f4febc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Getafix.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-01-12T11:14:02,clock=vm,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/aefd5844-6e01-4070-b3b9-c0d73cc40c78/52678e67-a202-4306-b7ed-5fed8df10edf/images/28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5/4c7b571e-4b29-47b9-ab4b-5799d64f28f9,if=none,id=drive-virtio-disk0,format=raw,serial=28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:74:59:a2,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0 -vnc 172.17.6.14:7,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on Qemu version: qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 Please let me know if I can do more to help! Thank you Martijn, Just curious, when opening Device Manager dialog, do you see High precision event timer device under System devices category? Best regards, Vadim. Best regards, Martijn. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On Mon, 2015-05-04 at 08:28 -0400, Michal Skrivanek wrote: On 4 May 2015, at 09:35, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. [snip] Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Sure! Command line is this: /usr/libexec/qemu-kvm -name Getafix -S -M rhel6.5.0 -cpu Penryn,hv_relaxed -enable-kvm -m 2048 -realtime mlock=off -smp 2,maxcpus=16,sockets=16,cores=1,threads=1 -uuid 34951c25-9a37-4712-a16a-fdfc98f4febc -smbios type=1,manufacturer=oVirt,product=oVirt Node,version=6-6.el6.centos.12.2,serial=44454C4C-3400-1058-804C-B1C04F42344A,uuid=34951c25-9a37-4712-a16a-fdfc98f4febc -nodefconfig -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Getafix.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=2015-01-12T11:14:02,clock=vm,driftfix=slew -no-kvm-pit-reinjection -no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -device virtio-serial-pci,id=virtio-serial0,max_ports=16,bus=pci.0,addr=0x4 -drive if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw,serial= -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/rhev/data-center/aefd5844-6e01-4070-b3b9-c0d73cc40c78/52678e67-a202-4306-b7e d-5fed8df10edf/images/28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5/4c7b571e-4b29-47b9-ab4b-5799d64f28f9,if=none,id=drive-virtio-disk0,format=raw,serial=28cc9a6c-6f2e-4b09-b361-f2a09f27dbc5,cache=none,werror=stop,rerror=stop,aio=threads -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -netdev tap,fd=41,id=hostnet0,vhost=on,vhostfd=43 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=00:1a:4a:74:59:a2,bus=pci.0,addr=0x3 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.com.redhat.rhevm.vdsm,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.rhevm.vdsm -chardev socket,id=charchannel1,path=/var/lib/libvirt/qemu/channels/34951c25-9a37-4712-a16a-fdfc98f4febc.org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=org.qemu.guest_agent.0 -device usb-table t,id=input0 -vnc 172.17.6.14:7,password -k en-us -vga cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6 -msg timestamp=on Qemu version: qemu-kvm-rhev-0.12.1.2-2.415.el6_5.14.x86_64 Please let me know if I can do more to help! How about trying virtio-scsi? Same difference? We'll be supporting virtio blk dataplane in 3.6, that may affect the performance significantly. Also a EL7 hypervisor could change results a lot. Do you have any at hand to give it a try? Well, also hyperv enlightnment? Not sure, but worth a try. It's currently disabled in osinfo entry for Win8/2012, can you try that?(on a non-production VM;) It worth trying, especially hv_time flag. Vadim. Thanks, michal Best regards, Martijn. ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
Hi Doron, I've also noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On Sun, 2015-05-03 at 07:46 -0500, Sven Achtelik wrote: Hi Vadim, I've tested the performance with CrystalDiskMark from inside the Windows guest. Using Win2k8 R2 I got expected values for my system, about 88 MB/s on 4k random with 32 queues and 500MB/s + sequential writes with 32 queues. Using a Windows 2012 VM on the same system it's only 33MB/s on 4k random with 32 queues and 300MB/s sequential writes. Similar tests with a linux VM show a bit better values than the Win2k8 R2 and respond ultra-fast. My hosts are connected via iSCSI using a 10 GbE link and a ZFS appliance as the storage system. All tests have been run several times with the same results. Sven, Can I ask you to post the Windows 2012 VM qemu command line? Thanks, Vadim. Sven -Ursprüngliche Nachricht- Von: Vadim Rozenfeld [mailto:vroze...@redhat.com] Gesendet: Sonntag, 3. Mai 2015 14:35 An: Sven Achtelik Cc: Doron Fediuck; Martijn Grendelman; Karen Noel; users@ovirt.org Betreff: Re: AW: Bad performance with Windows 2012 guests On Sun, 2015-05-03 at 06:48 -0500, Sven Achtelik wrote: Hi Doron, I've also noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On Sun, 2015-05-03 at 06:48 -0500, Sven Achtelik wrote: Hi Doron, I've also noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
Hi Vadim, I've tested the performance with CrystalDiskMark from inside the Windows guest. Using Win2k8 R2 I got expected values for my system, about 88 MB/s on 4k random with 32 queues and 500MB/s + sequential writes with 32 queues. Using a Windows 2012 VM on the same system it's only 33MB/s on 4k random with 32 queues and 300MB/s sequential writes. Similar tests with a linux VM show a bit better values than the Win2k8 R2 and respond ultra-fast. My hosts are connected via iSCSI using a 10 GbE link and a ZFS appliance as the storage system. All tests have been run several times with the same results. Sven -Ursprüngliche Nachricht- Von: Vadim Rozenfeld [mailto:vroze...@redhat.com] Gesendet: Sonntag, 3. Mai 2015 14:35 An: Sven Achtelik Cc: Doron Fediuck; Martijn Grendelman; Karen Noel; users@ovirt.org Betreff: Re: AW: Bad performance with Windows 2012 guests On Sun, 2015-05-03 at 06:48 -0500, Sven Achtelik wrote: Hi Doron, I've also noticed that there seems to be a difference in performance between Win2k8 R2/Linux and Windows Server 2012. After reading Martijns post I've done some speed test regarding the drive speeds and was looking for a way to compare the VMs on a more professional way. My tests showed, that on the same hardware, the Win2k8 R2 was faster in response and throughput on the disks. I found a utility that somehow measures the latency on a system and that also showed a significant difference. What is the correct way to do a performance test on a VM running in KVM ? Sven Hi Sven, Can you specify the type of disk on your system - ide, virtio-blk, or virtio-scsi? We usually use iometer for disk performance profiling. Sequential read/write, block size from 4K up to 256K, queue depth from 1 to 64. Vadim. -Ursprüngliche Nachricht- Von: users-boun...@ovirt.org [mailto:users-boun...@ovirt.org] Im Auftrag von Doron Fediuck Gesendet: Samstag, 2. Mai 2015 10:50 An: Martijn Grendelman Cc: Karen Noel; users@ovirt.org; Vadim Rozenfeld Betreff: Re: [ovirt-users] Bad performance with Windows 2012 guests On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
Re: [ovirt-users] Bad performance with Windows 2012 guests
On Apr 30, 2015 14:03, Martijn Grendelman martijn.grendel...@isaac.nl wrote: Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman Hi Martijn, Can you please provide the QEMU command line, together with kvm and qemu version? This information will be helpful for reproducing the problem. However, if the problem is not reproducible on a local setup, we will probably need to ask collecting some performance information with xperf tool. Doron ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
[ovirt-users] Bad performance with Windows 2012 guests
Hi, Ever since our first Windows Server 2012 deployment on oVirt (3.4 back then, now 3.5.1), I have noticed that working on these VMs via RDP or on the console via VNC is noticeably slower than on Windows 2008 guests on the same oVirt environment. Basic things like starting an application (even the Server Manager that get started automatically on login) take a very long time, sometimes minutes. Everything is just... slow. We have recently deployed Microsoft Exchange on a Windows Server 2012 guest on RHEV, and it doesn't perform well at all. I haven't been able to find the cause for this slowness; CPU usage is not excessive and it doesn't seem I/O related. Moreover, other types of guests (Linux and even Windows 2008) do not have this problem. We have 3 different environments: - oVirt 3.5.1, on old Dell servers with Penryn Family CPUs with fairly slow storage on replicated GlusterFS, running CentOS 6.6 - oVirt 3.5.1, on modern 6-core SandyBridge servers with local storage via NFS, running CentOS 7.0) - RHEV 3.4.4 on modern 10-core SandyBridge servers with an iSCSI SAN behind it, running on RHEV Hypervisor 6.5 All of these -very different- environments expose the same behaviour: Linux, Windows 2008 fast (or as fast as can be expected given the hardware), Windows 2012 painfully slow. All Windows 2012 servers use VirtIO disk and network. I think all drivers are from the virtio-win-0.1-74 ISO. Does anyone share this experience? Any idea why this could happen and how it can be fixed? Any other information I should share to get a better idea? Btw, for the guests on the RHEV environment, we have a case with RedHat support, but that doesn't seem to lead to a quick solution, hence I'm writing here, too. Thanks for any help. Regards, Martijn Grendelman ___ Users mailing list Users@ovirt.org http://lists.ovirt.org/mailman/listinfo/users
