Hi Groete, The VM virtual NIC network throttling is picked up from its compute offering. You may need to create a new compute offering and change the VM’s compute offering. If it is not specified in the compute offering, means it is taking the values from the global settings: vm.network.throttling.rate .
-Jithin From: Levin Ng <levindec...@gmail.com> Date: Sunday, 9 July 2023 at 5:00 AM To: users@cloudstack.apache.org <users@cloudstack.apache.org>, Granwille Strauss <granwi...@namhost.com> Cc: vivek.ku...@indiqus.com <vivek.ku...@indiqus.com>, Nux <n...@li.nux.ro> Subject: Re: Write Speeds Dear Groete, https://github.com/shapeblue/cloudstack/blob/965856057d5147f12b86abe5c9c205cdc5e44615/plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/DirectVifDriver.java https://libvirt.org/formatnetwork.html#quality-of-service This is in kilobytes/second, u have to divide by 8 1Gbps / 8bit = 128MB = 128000KBps You can verify by iperf test, and yes, u need to ensure both VR and VM match bandwidth settings to get a consistent result, something u also need to pay attention on VR resource, default system router resource offering is quite limited, the network speed may throttled if VR running are out of CPU resource. Regards, Levin On 8 Jul 2023 at 09:02 +0100, Granwille Strauss <granwi...@namhost.com>, wrote: > Hi Levin > Thank you very much, I do appreciate your feedback and time replying back to > me. I believe I have picked up on something. My VMs XML dump ALL show the > following: > > <interface type='bridge'> <mac address='xxxxxxx'/> <source > > bridge='cloudbr0'/> <bandwidth> <inbound average='128000' peak='128000'/> > > <outbound average='128000' peak='128000'/> </bandwidth> <target > > dev='vnet219'/> <model type='virtio'/> <link state='up'/> <alias > > name='net0'/> <rom bar='off' file=''/> <address type='pci' domain='0x0000' > > bus='0x00' slot='0x03' function='0x0'/> </interface> > Specifically: > <bandwidth> <inbound average='128000' peak='128000'/> <outbound > average='128000' peak='128000'/> </bandwidth> > All my VM has this in place, a 128 Mbps limit even the VR, which makes no > sense. I found this thread posted 10 years ago and Nux says that this value > is affected by the service offerings: > https://users.cloudstack.apache.narkive.com/T6Gx7BoV/cloudstack-network-limitation > But all my service offerings are set to 1000 Mbps. See attached screenshots. > The 4.18 documentation also confirms if the values are null, which most > default service offerings are, it takes the values set in global settings > network.throttling.rate and vm.network.throttling.rate, which I also have set > as 1000 as you can see in the screenshots. > > I then found this: > https://cwiki.apache.org/confluence/display/CLOUDSTACK/Network+throttling+in+CloudStack > with no KVM details, seems this part is missing to tell me how KVM > throttling is applied. So as DuJun said 10 years ago, I feel confused about > how Cloudstack limit the network rate for guest. And yes, I have stopped my > VMs and rebooted MANY times it doesn't update the XML at all. > Please also take into account documentation states that in shared networking > there's supposed to be no limits on incoming traffic (ingress), as far as I > understand it. > On 7/7/23 23:35, Levin Ng wrote: > > Hi Groete, > > > > > > IMO, You should bypass any ACS provisioning to troubleshoot the performance > > case first, which allow you get more idea on the hardware + kvm performance > > with minimal influent, then you can compare the libvirt xml different > > between plain KVM and ACS. That help you sort out the different where it > > come from, you will see QoS bandwidth setting in the VM xml if you do. > > > > We are trying to tell you, when you diagnose the throughput problem, you > > should first identify the bottleneck where it come first. Iperf is a tools > > that you can test the line speed end to end into your VM, if the result in > > 1Gbps network are near 800+ Mbps, you can focus on the VM performance or > > the copy protocol you are using, try different protocol, ssh/rsync/ftp/nfs, > > see any different. > > > > You are already test the write-back caching which will improve disk I/O > > performance, it is another story you need to deep dive the pro and cons on > > the write cache, there are risk to corrupt the VM filesystem in some case, > > this is what u need to learn about each cache mode. > > > > VM Guest performance are involved by many factor, you cannot expect VM > > perform nearly the bare metal does. There are long journey to do such > > optimization, take time and improve it gradually. There are lot of kvm > > tuning guide you can reference and prove it on your hardware. Read > > thoughtfully on each tuning that may bring improvement and also introduce > > risk factor. > > > > > > Regards, > > Levin > > > > > > > > > > On 7 Jul 2023 at 21:24 +0100, Granwille Strauss <granwi...@namhost.com>, > > wrote: > > > Sorry that I have to ask, can you perhaps be a bit more specific, please. > > > The only QOS settings I see in service offering are "None", "Hypervisor" > > > and "Storage", which doesn't really seem network related. Or am I missing > > > the point? Take note that I use the default offerings for the VR and VMs > > > but with slight tweaks such as setting local storage etc and only > > > increased the Network rate from 200 Mbps to 1000 Mbps. > > > > > > So can you kindly explain by what QOS settings you guys are referring to, > > > please? > > > PS, the Write-back disk caching seems to give the VM a slight increase, I > > > now see writes at 190 Mbps from ~70 Mbps. > > > On 7/7/23 21:11, Vivek Kumar wrote: > > > > Hello, > > > > > > > > IPerf will simply tell you the bandwidth and the open pipe between 2 > > > > VMs, so I don’t think that it’s depends on disk performance, it’s > > > > better to check the network QoS at every layer, VR and VM. > > > > > > > > > > > > > > > > Vivek Kumar > > > > Sr. Manager - Cloud & DevOps > > > > TechOps | Indiqus Technologies > > > > > > > > vivek.ku...@indiqus.com <mailto:vivek.ku...@indiqus.com> > > > > www.indiqus.com<http://www.indiqus.com> <https://www.indiqus.com/> > > > > > > > > > > > > > > > > > > > > > On 07-Jul-2023, at 9:44 PM, Granwille Strauss <granwi...@namhost.com.INVALID> > wrote: > > > > > > > > > > Hi Levin > > > > > > > > > > Thank you, I am aware of network offering, the first thing I did was > > > > > make sure it was set to accommodate the KVM's entire 1 Gbps uplink. > > > > > But now that I think if it iperf test prevousily were always stuck on > > > > > 50 Mbps, but this is because of the write speeds on the disk at least > > > > > that's what I believe causes the network bottle neck. I will > > > > > double-check this again. > > > > > > > > > > But there is some sort of limit on the VM disk in place. FIO tests > > > > > show that write speeds are in the range of 50 - 90 MB/s on the VM, > > > > > while fio test confirms on the KVM its over 400 MB/s. > > > > > > > > > > On 7/7/23 18:08, Levin Ng wrote: > > > > > > Hi Groete, > > > > > > > > > > > > Forgot to mention, when you are talking about file copies between > > > > > > remote server, you need to aware there are network QoS option in > > > > > > the offering, make sure the limits correctness. Do iperf test prove > > > > > > that too, test between server and via virtual router. Hope you can > > > > > > narrow down the problem soon. > > > > > > > > > > > > Regards, > > > > > > Levin > > > > > > > > > > > > On 7 Jul 2023 at 16:40 +0100, Granwille Strauss > > > > > > <granwi...@namhost.com> <mailto:granwi...@namhost.com>, wrote: > > > > > > > Hi Levin > > > > > > > Thank you, yes I leave IOPs empty. And the KVM host has SSDs in a > > > > > > > hardware RAID 5 configuration, of which I am using local storage > > > > > > > pool, yes. I will run fio test and also playing around with the > > > > > > > controller cache settings to see what happens and provide > > > > > > > feedback on this soon. > > > > > > > On 7/7/23 17:23, Levin Ng wrote: > > > > > > > > HI Groete, > > > > > > > > > > > > > > > > Should you run a fio test on the VM and the KVM host to get a > > > > > > > > baseline first. SSD are tricky device, when it fill up the > > > > > > > > cache or nearly full, the performance will drop significantly, > > > > > > > > especially consumer grade SSD. There are option to limit IOPs > > > > > > > > in ACS offering setting, I believe you leave it empty, so it is > > > > > > > > no limit. When you talking about KVM uses SSDs, I think you are > > > > > > > > using Local Disk Pool right? If you have RAID controller > > > > > > > > underlying, try toggle the controller cache, SSD may perform > > > > > > > > vary on different disk controller cache setting. > > > > > > > > > > > > > > > > Controller type scsi, or virtio performance are similar, no > > > > > > > > need to worry about it. Of coz, in general, using RAW format > > > > > > > > and thick provisioning could get a best io performance result, > > > > > > > > but consume space and lack of snapshot capabliblity , so most > > > > > > > > the time it is not prefer go this path. > > > > > > > > > > > > > > > > Please gather more information first > > > > > > > > > > > > > > > > Regards, > > > > > > > > Levin > > > > > > > > On 7 Jul 2023 at 15:30 +0100, Granwille Strauss > > > > > > > > <granwi...@namhost.com.invalid> > > > > > > > > <mailto:granwi...@namhost.com.invalid>, wrote: > > > > > > > > > Hi Guys > > > > > > > > > Does Cloudstack have a disk write speed limit somewhere in > > > > > > > > > its setting? We have been transferring many files from remote > > > > > > > > > servers to VM machines on our Cloudstack instance and we > > > > > > > > > recently noticed that the VM write speeds are all limited to > > > > > > > > > about 5-8 MB/s. But the underlying hardware of the KVM uses > > > > > > > > > SSDs capable of write speeds of 300 - 600 MB/s. My disk > > > > > > > > > offering on my current vms are set to "No Disk Cache" with > > > > > > > > > thin provisioning, could this be the reason? I understand > > > > > > > > > that "Write Back Disk Cach" has better write speeds. Also I > > > > > > > > > have VMs set as virtio for its disk controller. What could I > > > > > > > > > be missing in this case? > > > > > > > > > -- > > > > > > > > > Regards / Groete > > > > > > > > > > > > > > > > > > Granwille Strauss // Senior Systems Admin > > > > > > > > > > > > > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > > > > > > > m: +264 81 323 1260 > > > > > > > > > w: www.namhost.com<http://www.namhost.com> > > > > > > > > > <http://www.namhost.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Namhost Internet Services (Pty) Ltd, > > > > > > > > > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > > > > > > > > > > > > > > > > > > > > > > > > > The content of this message is confidential. If you have > > > > > > > > > received it by mistake, please inform us by email reply and > > > > > > > > > then delete the message. It is forbidden to copy, forward, or > > > > > > > > > in any way reveal the contents of this message to anyone > > > > > > > > > without our explicit consent. The integrity and security of > > > > > > > > > this email cannot be guaranteed over the Internet. Therefore, > > > > > > > > > the sender will not be held liable for any damage caused by > > > > > > > > > the message. For our full privacy policy and disclaimers, > > > > > > > > > please go to https://www.namhost.com/privacy-policy > > > > > > > > > > > > > > > > -- > > > > > > > Regards / Groete > > > > > > > > > > > > > > Granwille Strauss // Senior Systems Admin > > > > > > > > > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > > > > > m: +264 81 323 1260 > > > > > > > w: www.namhost.com<http://www.namhost.com> > > > > > > > <http://www.namhost.com/> > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Namhost Internet Services (Pty) Ltd, > > > > > > > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > > > > > > > > > > > > > > > > > > > The content of this message is confidential. If you have received > > > > > > > it by mistake, please inform us by email reply and then delete > > > > > > > the message. It is forbidden to copy, forward, or in any way > > > > > > > reveal the contents of this message to anyone without our > > > > > > > explicit consent. The integrity and security of this email cannot > > > > > > > be guaranteed over the Internet. Therefore, the sender will not > > > > > > > be held liable for any damage caused by the message. For our full > > > > > > > privacy policy and disclaimers, please go to > > > > > > > https://www.namhost.com/privacy-policy > > > > > > > > > > > > -- > > > > > Regards / Groete > > > > > > > > > > <https://www.namhost.com/> Granwille Strauss // Senior Systems > > > > > Admin > > > > > > > > > > e: granwi...@namhost.com <mailto:granwi...@namhost.com> > > > > > m: +264 81 323 1260 <tel:+264813231260> > > > > > w: www.namhost.com<http://www.namhost.com> <https://www.namhost.com/> > > > > > > > > > > <https://www.facebook.com/namhost> <https://twitter.com/namhost> > > > > > <https://www.instagram.com/namhostinternetservices/> > > > > > <https://www.linkedin.com/company/namhos> > > > > > <https://www.youtube.com/channel/UCTd5v-kVPaic_dguGur15AA> > > > > > > > > > > <https://www.adsigner.com/v1/l/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818/banner> > > > > > > > > > > Namhost Internet Services (Pty) Ltd, 24 Black Eagle Rd, Hermanus, > > > > > 7210, RSA > > > > > > > > > > The content of this message is confidential. If you have received it > > > > > by mistake, please inform us by email reply and then delete the > > > > > message. It is forbidden to copy, forward, or in any way reveal the > > > > > contents of this message to anyone without our explicit consent. The > > > > > integrity and security of this email cannot be guaranteed over the > > > > > Internet. Therefore, the sender will not be held liable for any > > > > > damage caused by the message. For our full privacy policy and > > > > > disclaimers, please go to https://www.namhost.com/privacy-policy > > > > > <https://www.adsigner.com/v1/c/631091998d4670001fe43ec2/621c9b76c140bb001ed0f818> > > > -- > > > Regards / Groete > > > > > > Granwille Strauss // Senior Systems Admin > > > > > > e: granwi...@namhost.com > > > m: +264 81 323 1260 > > > w: www.namhost.com<http://www.namhost.com> > > > > > > > > > > > > > > > Namhost Internet Services (Pty) Ltd, > > > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > > > > > > > The content of this message is confidential. If you have received it by > > > mistake, please inform us by email reply and then delete the message. It > > > is forbidden to copy, forward, or in any way reveal the contents of this > > > message to anyone without our explicit consent. The integrity and > > > security of this email cannot be guaranteed over the Internet. Therefore, > > > the sender will not be held liable for any damage caused by the message. > > > For our full privacy policy and disclaimers, please go to > > > https://www.namhost.com/privacy-policy > > > > -- > Regards / Groete > > Granwille Strauss // Senior Systems Admin > > e: granwi...@namhost.com > m: +264 81 323 1260 > w: www.namhost.com<http://www.namhost.com> > > > > > Namhost Internet Services (Pty) Ltd, > 24 Black Eagle Rd, Hermanus, 7210, RSA > > > The content of this message is confidential. If you have received it by > mistake, please inform us by email reply and then delete the message. It is > forbidden to copy, forward, or in any way reveal the contents of this message > to anyone without our explicit consent. The integrity and security of this > email cannot be guaranteed over the Internet. Therefore, the sender will not > be held liable for any damage caused by the message. For our full privacy > policy and disclaimers, please go to https://www.namhost.com/privacy-policy >
