I'm afraid I don't follow. I was talking about multiple core VMs and not single core ones. I said if I have a 4 core VM with 3Ghz core speed and I run 4 cpuburn threads I could get approximately the performance of a single core at 12Ghz.
-----Original Message----- From: Edison Su [mailto:[email protected]] Sent: 23 May 2012 23:06 To: [email protected] Subject: RE: Vmware CPU Cap Single core means there is only one scheduling entity, that hypervisor can schedule for the VM, so there is no parallel processing at all. Hypervisor can not schedule a single core VM to multiple physical cpu cores at the same time. > -----Original Message----- > From: Tamas Monos [mailto:[email protected]] > Sent: Wednesday, May 23, 2012 2:59 PM > To: [email protected] > Subject: RE: Vmware CPU Cap > > Hi, > > Before we would touch that rule 2: > While utilizing 12Ghz on a single core is not possible it is possible > to have effective 12Ghz on 4 cores at 3Ghz via parallel processing. > If I start 4 cpuburn threads on a quad core it will use all four 3Ghz > cores flat out "giving similar speed as 1 core@12Ghz would be"*. As > vmware uses the limit parameter as limit to the whole CPU utilization > rule 2 approach would prevent me effectively utilizing the CPU on > vmware because if we say the CPU speed in Mhz cannot be larger than > the core speed I would be limited to an effective speed of core- > speed/number_of_cores. This would mean 3000/4=750Mhz per core on a > 4vCPU VM. If you set "core_number*core_speed" as limit you already > broke rule 2 as the CPU is set higher than its core speed. > > I believe it good as it is now. > > Regards > > -----Original Message----- > From: Prachi Damle [mailto:[email protected]] > Sent: 23 May 2012 22:36 > To: [email protected] > Subject: RE: Vmware CPU Cap > > Yeah, rule#1 and #3 are considered by the current Host Allocator; > rule#2 to compare just the speed is missing. > > Prachi > > -----Original Message----- > From: Anthony Xu [mailto:[email protected]] > Sent: Wednesday, May 23, 2012 2:06 PM > To: [email protected] > Subject: RE: Vmware CPU Cap > > Checked the code, I think we miss rule 2 in allocator. > > Anthony > > > -----Original Message----- > > From: Anthony Xu [mailto:[email protected]] > > Sent: Wednesday, May 23, 2012 1:50 PM > > To: [email protected] > > Subject: RE: Vmware CPU Cap > > > > I'm pretty sure we have that before we redesign allocator. > > > > Anthony > > > > > -----Original Message----- > > > From: Alex Huang [mailto:[email protected]] > > > Sent: Wednesday, May 23, 2012 1:44 PM > > > To: [email protected] > > > Subject: RE: Vmware CPU Cap > > > > > > I'm surprised. I definitely recall having that conversation > > > before about never exceeding the physical limits of the > > > hypervisor. I'm pretty sure we have test cases against that as well. > > > > > > --Alex > > > > > > > -----Original Message----- > > > > From: Edison Su [mailto:[email protected]] > > > > Sent: Wednesday, May 23, 2012 1:39 PM > > > > To: [email protected] > > > > Subject: RE: Vmware CPU Cap > > > > > > > > I think it's a general issue for all the hypervisors we supported. > > > > > > > > > -----Original Message----- > > > > > From: Alex Huang [mailto:[email protected]] > > > > > Sent: Wednesday, May 23, 2012 1:32 PM > > > > > To: [email protected] > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > That's because no one wrote a host allocator for VmWare. > > > > > > > > > > The relationship between DeploymentPlanner and HostAllocator > and > > > > > StoragePoolAllocator is as follows: > > > > > > > > > > DeploymentPlanner deals with heuristics and affinity rules of > > > placing > > > > > a VM. Once it determines a set of hosts that matches, it then > > asks > > > > > the HostAllocator to work on the limitations of the type of > > > hypervisor. > > > > > There was never a HostAllocator written for VmWare. It just > > > > > uses > > > the > > > > > one that was written originally for XenServer. Hence the bug. > > > > > > > > > > There's similar differentiation for DeploymentPlanner and > > > > > StoragePoolAllocator and similar bugs exists, especially if > > someone > > > > > adds a completely new type of storage pool but decides to just > > > reuse > > > > > the current set of StoragePoolAllocators. > > > > > > > > > > --Alex > > > > > > > > > > > -----Original Message----- > > > > > > From: Edison Su [mailto:[email protected]] > > > > > > Sent: Wednesday, May 23, 2012 1:24 PM > > > > > > To: [email protected]; '[email protected]' > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > Thanks for your input, I find another bug in cloudstack... > > > > > > Let's dive little bit deeper into how the CPU allocation > works. > > > > > > VMware(the same for other hypervisors(Xen/KVM)) uses > > proportional > > > > > share > > > > > > based scheduling algorithm([1],[2]). It means "If a virtual > > > machine > > > > > has twice > > > > > > as many shares of a resource as another virtual machine, it > is > > > > > entitled to > > > > > > consume twice as much of that resource when these two > > > > > > virtual > > > > > machines > > > > > > are competing for resources." > > > > > > There are two questions: > > > > > > 1. Where is the share coming from? In Cloudstack, the share > is > > > > > calculated > > > > > > from (CPU MHz * CPU cores). > > > > > > 2. How the share of a VM is mapped to the physical CPU by > > > > > > the > > > > > hypervisor? > > > > > > Of cause, it depends on the hypervisor implementation, but > > there > > > are > > > > > some > > > > > > general rules that we need to follow, as we are living in > > > > > > the > > > real > > > > > world:) > > > > > > Rule 1: num of cpu core of a VM should be <= num of cpu > > > > > > core > > > of > > > > > the > > > > > > hypervisor host. It doesn't make sense to running a 12 core > VM > > on > > > a > > > > > single > > > > > > core host, even some hypervisors, such as KVM, can do that. > > > > > Hypervisor will > > > > > > be busy at scheduling VCPU(the VCPU context switch is much > > > heavier > > > > > than > > > > > > process context switch), thus bad performance for the VM. > > > > > > Rule 2: The frequency of a VCPU should be <= the > > > > > > frequency > > of > > > the > > > > > host > > > > > > hypervisor host. Can you run an one core * 12Ghz VM on a > > > > > > 3Ghz > > > > > > * > > 4 > > > > > core > > > > > > physical CPU? Nope, in an unit time, the max freq of VCPU > > > > > > can > > get > > > is > > > > > the max > > > > > > freq of physical cpu core, that's the physical LAW. > > > > > > Rule 3: the total share of VMs <= total share of host > > > > > hypervisor(in case of > > > > > > no CPU overcommit). > > > > > > > > > > > > Currently, in cloudstack host allocator, only rule 3 is > > > > > > taken > > > into > > > > > consideration. > > > > > > > > > > > > [1] www.cs.uiuc.edu/class/sp10/cs423/lectures/17-VMM- > > resource.pdf > > > > > > [2] > > > > > > > > > > http://www.vmware.com/pdf/vsphere4/r40/vsp_40_resource_mgmt.pdf > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Tamas Monos [mailto:[email protected]] > > > > > > > Sent: Tuesday, May 22, 2012 5:41 PM > > > > > > > To: [email protected] > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I still don't think this is an issue as the CPU Mhz limit > > > > > > > and > > > the > > > > > > > number of cores are independent. > > > > > > > CPU manufacturers sell 2,4,6 cores at 3Ghz and not > > > > > > > 6,12,18Ghz > > > CPUs. > > > > > > > > > > > > > > So I think it is good how it works but the > > > "number_of_cores*Mhz" > > > > > while > > > > > > > allocating should not multiply so that is the bug :) What > > > vmware > > > > > does > > > > > > > with multiplying the cores with the core speed is bad as I > > > can't > > > > > have > > > > > > > a 1vCPU VM at 12Ghz on a 3Ghz quad core if you know where > > > > > > > I'm > > > > > coming > > > > > > > from.... > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Tamas Monos [mailto:[email protected]] > > > > > > > Sent: 23 May 2012 01:27 > > > > > > > To: [email protected] > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I have confused myself too because if I have a look the > > > database > > > > > and > > > > > > > dump the service_offerings table the limit for me is set > > > > > > > to > > > > > > > 2000,3000,4000 and not 1000. > > > > > > > So a Mhz limit set to 4000 and 4 cores will end up as a > quad > > > core > > > > > box > > > > > > > at 4Ghz. I remember now I had to set CPU overprovisioning > to > > 4 > > > as > > > > > > > CloudStack took away 4x4Ghz=16Ghz of the available CPU. > > > > > > > So Cloudstack sets the VM CPU Mhz limit what the actual > > > > > > > limit > > > is > > > > > set > > > > > > > to in the offering but does not multiply the Mhz limit by > > > > > > > the > > > > > number > > > > > > > of cores when setting the limit on vmware. However takes > > > > > > > away "number_of_cores*Mhz limit" from the available CPU > > > > > > > capacity > > > when > > > > > > > allocating. > > > > > > > > > > > > > > I'm getting confused myself so I'm not sure if this is bug > > now > > > or > > > > > not > > > > > > > either :) I'm using 3.0.3 > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Edison Su [mailto:[email protected]] > > > > > > > Sent: 23 May 2012 00:32 > > > > > > > To: [email protected]; Tamas Monos > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > But in Diego'case, the limit is set as 1000Mhz, while his > > > service > > > > > > > offering is 1000Mhz * 2 cores. > > > > > > > Which version of cloudstack are you using? Maybe it's a > > > regression. > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Tamas Monos [mailto:[email protected]] > > > > > > > > Sent: Tuesday, May 22, 2012 4:26 PM > > > > > > > > To: [email protected] > > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I have a service offering with 1000Mhz limit and 4 cpu > > cores. > > > > > That > > > > > > > > sums up to 4000Mhz. > > > > > > > > Cloudstack sets the limit to 4Ghz on the virtual machine > > and > > > > > > > > when > > > > > I > > > > > > > > put load on it vmware balances the load between the 4 > > > > > > > > cores > > > > > allowing > > > > > > > > them to use 1000Mhz each. > > > > > > > > I do not see any bugs here. > > > > > > > > > > > > > > > > Apologies if the "meant per CPU core" was incorrect. > > > > > > > > What I meant > > > > > is > > > > > > > > described above. > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > From: Edison Su [mailto:[email protected]] > > > > > > > > Sent: 23 May 2012 00:05 > > > > > > > > To: [email protected] > > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > > > Nope, from the document, the limit is set on whole vm: > > > > > > > > CPU Limits > > > > > > > > > > > > > > > > When a CPU Limit is set on a virtual machine resource > > > settings, > > > > > the > > > > > > > > virtual machine is deliberately held from being > > > > > > > > scheduled > > to > > > a > > > > > PCPU > > > > > > > > when it has used up its allocated CPU resource. This > > happens > > > > > > > > regardless of the CPU utilization. If the limit is set > > > > > > > > to 500MHz, the virtual machine is descheduled from the > > > > > > > > PCPU > > and > > > has > > > > > > > > to wait before > > > > > > > it > > > > > > > > is allowed to be scheduled again. As such, the virtual > > > machine > > > > > might > > > > > > > > experience performance degradation. > > > > > > > > > > > > > > > > Note: For an SMP virtual machine, the sum of all vCPUs > > cannot > > > > > exceed > > > > > > > > the specified limit. For example, 4 vCPU virtual machine > > with > > > a > > > > > > > > limit of 1200MHz and equal load among vCPUs would result > > > > > > > > in > > a > > > > > > > > max > > > > > of > > > > > > > > 300MHz per vCPU. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > http://kb.vmware.com/selfservice/documentLinkInt.do?micrositeID=&pop > > > > > > u > > > > > > p > > > > > > > > = > > > > > > > > true&languageId=&externalID=1033115 > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Tamas Monos [mailto:[email protected]] > > > > > > > > > Sent: Tuesday, May 22, 2012 3:43 PM > > > > > > > > > To: [email protected] > > > > > > > > > Subject: RE: Vmware CPU Cap > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > No I don't think this is a bug. When you set 1000Mhz > > > > > > > > > as > > CPU > > > > > > > > > cap > > > > > > > that > > > > > > > > > is meant per core. vmWare will limit each CPU core to > > > 1000Mhz. > > > > > > > > > As you gave 2 CPU cores that is 2000Mhz effective. > > > > > > > > > That > > is > > > how > > > > > > > > > vmware works. > > > > > > > > > > > > > > > > > > I have setup my offerings all to 1000Mhz as speed and > > just > > > > > > > > > increasing the number of cores. > > > > > > > > > 1 core ends up being 1x1000Mhz > > > > > > > > > 2 core ends up being 2x1000Mhz=2000Mhz ... > > > > > > > > > ... > > > > > > > > > > > > > > > > > > I'm actually using it in production and works quite > well > > as > > > > > > > > Cloudstack > > > > > > > > > "allocates" 4000Mhz when I'm using 4x1000Mhz cores and > > > vmware > > > > > > > > cleverly > > > > > > > > > balances between the cores as you put load on it and > not > > > > > letting > > > > > > > any > > > > > > > > > of the cores above the set limit of 1000Mhz. > > > > > > > > > > > > > > > > > > > > > > > > > > > Regards > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > > > From: Diego Spinola Castro > > [mailto:[email protected]] > > > > > > > > > Sent: 22 May 2012 19:34 > > > > > > > > > To: [email protected] > > > > > > > > > Subject: Re: Vmware CPU Cap > > > > > > > > > > > > > > > > > > I forgot the link: > > > > > > > > > http://imageshack.us/photo/my-images/848/cpulimit.png/ > > > > > > > > > > > > > > > > > > 2012/5/22 Diego Spinola Castro > <[email protected]> > > > > > > > > > > > > > > > > > > > I believe that is a bug with cpu cap and vmware. > > > > > > > > > > > > > > > > > > > > To reproduce: > > > > > > > > > > > > > > > > > > > > Create a offering with 2 cores and 1000mhz. > > > > > > > > > > Enable CPU CAP. > > > > > > > > > > > > > > > > > > > > After created instance , cs create a vm with 2 cores > > and > > > > > 1000mhz > > > > > > > > > > of > > > > > > > > > limit. > > > > > > > > > > > > > > > > > > > > I don't know for sure if is a bug, but vmware gives > > > 1000mhz > > > > > > > shared > > > > > > > > > > with cores. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Diego > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
