Hi Groete,
IMO, You should bypass any ACS provisioning to troubleshoot the performance case first, which allow you get more idea on the hardware + kvm performance with minimal influent, then you can compare the libvirt xml different between plain KVM and ACS. That help you sort out the different where it come from, you will see QoS bandwidth setting in the VM xml if you do. We are trying to tell you, when you diagnose the throughput problem, you should first identify the bottleneck where it come first. Iperf is a tools that you can test the line speed end to end into your VM, if the result in 1Gbps network are near 800+ Mbps, you can focus on the VM performance or the copy protocol you are using, try different protocol, ssh/rsync/ftp/nfs, see any different. You are already test the write-back caching which will improve disk I/O performance, it is another story you need to deep dive the pro and cons on the write cache, there are risk to corrupt the VM filesystem in some case, this is what u need to learn about each cache mode. VM Guest performance are involved by many factor, you cannot expect VM perform nearly the bare metal does. There are long journey to do such optimization, take time and improve it gradually. There are lot of kvm tuning guide you can reference and prove it on your hardware. Read thoughtfully on each tuning that may bring improvement and also introduce risk factor. Regards, Levin On 7 Jul 2023 at 21:24 +0100, Granwille Strauss <granwi...@namhost.com>, wrote: > Sorry that I have to ask, can you perhaps be a bit more specific, please. The > only QOS settings I see in service offering are "None", "Hypervisor" and > "Storage", which doesn't really seem network related. Or am I missing the > point? Take note that I use the default offerings for the VR and VMs but with > slight tweaks such as setting local storage etc and only increased the > Network rate from 200 Mbps to 1000 Mbps. > > So can you kindly explain by what QOS settings you guys are referring to, > please? > PS, the Write-back disk caching seems to give the VM a slight increase, I now > see writes at 190 Mbps from ~70 Mbps. > On 7/7/23 21:11, Vivek Kumar wrote: > > Hello, > > > > IPerf will simply tell you the bandwidth and the open pipe between 2 VMs, > > so I don’t think that it’s depends on disk performance, it’s better to > > check the network QoS at every layer, VR and VM. > > > > > > > > Vivek Kumar > > Sr. Manager - Cloud & DevOps > > TechOps | Indiqus Technologies > > > > vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> > > www.indiqus.com <https://www.indiqus.com/> > > > > > > > > > > > On 07-Jul-2023, at 9:44 PM, Granwille Strauss > > > <granwi...@namhost.com.INVALID> wrote: > > > > > > Hi Levin > > > > > > Thank you, I am aware of network offering, the first thing I did was make > > > sure it was set to accommodate the KVM's entire 1 Gbps uplink. But now > > > that I think if it iperf test prevousily were always stuck on 50 Mbps, > > > but this is because of the write speeds on the disk at least that's what > > > I believe causes the network bottle neck. I will double-check this again. > > > > > > But there is some sort of limit on the VM disk in place. FIO tests show > > > that write speeds are in the range of 50 - 90 MB/s on the VM, while fio > > > test confirms on the KVM its over 400 MB/s. > > > > > > On 7/7/23 18:08, Levin Ng wrote: > > > > Hi Groete, > > > > > > > > Forgot to mention, when you are talking about file copies between > > > > remote server, you need to aware there are network QoS option in the > > > > offering, make sure the limits correctness. Do iperf test prove that > > > > too, test between server and via virtual router. Hope you can narrow > > > > down the problem soon. > > > > > > > > Regards, > > > > Levin > > > > > > > > On 7 Jul 2023 at 16:40 +0100, Granwille Strauss <granwi...@namhost.com> > > > > <mailto:granwi...@namhost.com>, wrote: > > > > > Hi Levin > > > > > Thank you, yes I leave IOPs empty. And the KVM host has SSDs in a > > > > > hardware RAID 5 configuration, of which I am using local storage > > > > > pool, yes. I will run fio test and also playing around with the > > > > > controller cache settings to see what happens and provide feedback on > > > > > this soon. > > > > > On 7/7/23 17:23, Levin Ng wrote: > > > > > > HI Groete, > > > > > > > > > > > > Should you run a fio test on the VM and the KVM host to get a > > > > > > baseline first. SSD are tricky device, when it fill up the cache or > > > > > > nearly full, the performance will drop significantly, especially > > > > > > consumer grade SSD. There are option to limit IOPs in ACS offering > > > > > > setting, I believe you leave it empty, so it is no limit. When you > > > > > > talking about KVM uses SSDs, I think you are using Local Disk Pool > > > > > > right? If you have RAID controller underlying, try toggle the > > > > > > controller cache, SSD may perform vary on different disk controller > > > > > > cache setting. > > > > > > > > > > > > Controller type scsi, or virtio performance are similar, no need to > > > > > > worry about it. Of coz, in general, using RAW format and thick > > > > > > provisioning could get a best io performance result, but consume > > > > > > space and lack of snapshot capabliblity , so most the time it is > > > > > > not prefer go this path. > > > > > > > > > > > > Please gather more information first > > > > > > > > > > > > Regards, > > > > > > Levin > > > > > > On 7 Jul 2023 at 15:30 +0100, Granwille Strauss > > > > > > <granwi...@namhost.com.invalid> > > > > > > <mailto:granwi...@namhost.com.invalid>, wrote: > > > > > > > Hi Guys > > > > > > > Does Cloudstack have a disk write speed limit somewhere in its > > > > > > > setting? We have been transferring many files from remote servers > > > > > > > to VM machines on our Cloudstack instance and we recently noticed > > > > > > > that the VM write speeds are all limited to about 5-8 MB/s. But > > > > > > > the underlying hardware of the KVM uses SSDs capable of write > > > > > > > speeds of 300 - 600 MB/s. My disk offering on my current vms are > > > > > > > set to "No Disk Cache" with thin provisioning, could this be the > > > > > > > reason? I understand that "Write Back Disk Cach" has better write > > > > > > > speeds. Also I have VMs set as virtio for its disk controller. > > > > > > > What could I be missing in this case? > > > > > > > -- > > > > > > > Regards / Groete > > > > > > > > > > > > > > Granwille Strauss // Senior Systems Admin > > > > > > > > > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > > > > > m: +264 81 323 1260 > > > > > > > w: www.namhost.com <http://www.namhost.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Namhost Internet Services (Pty) Ltd, > > > > > > > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > > > > > > > > > > > > > > > > > > > The content of this message is confidential. If you have received > > > > > > > it by mistake, please inform us by email reply and then delete > > > > > > > the message. It is forbidden to copy, forward, or in any way > > > > > > > reveal the contents of this message to anyone without our > > > > > > > explicit consent. The integrity and security of this email cannot > > > > > > > be guaranteed over the Internet. Therefore, the sender will not > > > > > > > be held liable for any damage caused by the message. For our full > > > > > > > privacy policy and disclaimers, please go to > > > > > > > https://www.namhost.com/privacy-policy > > > > > > > > > > > > -- > > > > > Regards / Groete > > > > > > > > > > Granwille Strauss // Senior Systems Admin > > > > > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > > > m: +264 81 323 1260 > > > > > w: www.namhost.com <http://www.namhost.com/> > > > > > > > > > > > > > > > > > > > > > > > > > Namhost Internet Services (Pty) Ltd, > > > > > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > > > > > > > > > > > > > The content of this message is confidential. If you have received it > > > > > by mistake, please inform us by email reply and then delete the > > > > > message. It is forbidden to copy, forward, or in any way reveal the > > > > > contents of this message to anyone without our explicit consent. The > > > > > integrity and security of this email cannot be guaranteed over the > > > > > Internet. Therefore, the sender will not be held liable for any > > > > > damage caused by the message. For our full privacy policy and > > > > > disclaimers, please go to https://www.namhost.com/privacy-policy > > > > > > > > -- > > > Regards / Groete > > > > > > <https://www.namhost.com/> Granwille Strauss // Senior Systems Admin > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > m: +264 81 323 1260 <tel:+264813231260> > > > w: www.namhost.com <https://www.namhost.com/> > > > > > > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > > > <https://www.instagram.com/namhostinternetservices/> > > > <https://www.linkedin.com/company/namhos> > > > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > > > > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > > > > > Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, 7210, > > > RSA > > > > > > The content of this message is confidential. If you have received it by > > > mistake, please inform us by email reply and then delete the message. It > > > is forbidden to copy, forward, or in any way reveal the contents of this > > > message to anyone without our explicit consent. The integrity and > > > security of this email cannot be guaranteed over the Internet. Therefore, > > > the sender will not be held liable for any damage caused by the message. > > > For our full privacy policy and disclaimers, please go to > > > https://www.namhost.com/privacy-policy > > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > -- > Regards / Groete > > Granwille Strauss // Senior Systems Admin > > e: granwi...@namhost.com > m: +264 81 323 1260 > w: www.namhost.com > > > > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this message > to anyone without our explicit consent. The integrity and security of this > email cannot be guaranteed over the Internet. Therefore, the sender will not > be held liable for any damage caused by the message. For our full privacy > policy and disclaimers, please go to https://www.namhost.com/privacy-policy >
