On 6/5/12 4:17 PM, "Evan Miller" <evan.mil...@citrix.com> wrote:
>FYI. > >Alena manually removed the expunging instances from my CloudStack >Management Server. Small correction. I didn't manually removed the instances, I just triggered expunge thread to run every 60 seconds, and the vms got expunged by system following normal expunge process. The one thing I did via mysql db - set up expunge.delay/expunge.inteval to 60 in cloud.config table; but it's also possible to do on Configuration tab in cloudStack UI. I should have done it the official way rather then mocking up with the DB. -Alena. > As a result, two allocated ip addresses were >freed up. Then, I could add the VM instance without error (without >an insufficient resources error). Essentially, the default time >for the system to remove any expunging instances had not been >reached yet. The manual, removal steps involved executing MySQL >commands that updated a particular cloud table. > >This issue is resolved. Thanks, Alena. > >Regards, >Evan > >-----Original Message----- >From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com] >Sent: Tuesday, June 05, 2012 1:14 PM >To: cloudstack-users@incubator.apache.org >Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is >in avoid set) > >On 6/5/12 12:52 PM, "Evan Miller" <evan.mil...@citrix.com> wrote: > >>Attached is the job-18 log snippet. >>Sorry for not including this attachment with the previous email. >> >>Regards, >>Evan >>-----Original Message----- >>From: Evan Miller [mailto:evan.mil...@citrix.com] >>Sent: Tuesday, June 05, 2012 12:48 PM >>To: cloudstack-users@incubator.apache.org >>Subject: RE: Unable to add an instance - speed=500Mhz, Ram=512 (host is >>in avoid set) >> >> >>-----Original Message----- >>From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com] >>Sent: Tuesday, June 05, 2012 12:08 PM >>To: cloudstack-users@incubator.apache.org >>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host is >>in avoid set) >> >>On 6/5/12 11:49 AM, "Evan Miller" <evan.mil...@citrix.com> wrote: >> >>>RECOMMENDATION: >>>Need to improve the 3.0.0 - 3.0.2 Basic Installation Guide. >>>I don't have huge ranges of ip addresses available. So, it would be >>>extremely beneficial to have some idea, in advance, as to how ip >>>addresses will be consumed by a basic cloud setup. For example, I was >>>surprised to learn that in my basic setup, one CloudStack Management >>>Server (also serving primary and secondary filesystems) and one >>>XenServer using one connected nic, the following ip addresses were >>>required even before adding a single VM instance: >>> >>> CSMS - 1 ip (management) [not surprising] XenServer - 1 ip >>> (management) [not surprising] s-1-VM - 3 ips (2 - management, 1 - >>> guest) [surprising] v-2-VM - 2 ips (1 - management, 1 - guest) >>> [surprising] >>> >>>So, for each VM instance, is only one guest ip address required? Or is >>>more required? >> >>Only one guest IP. >> >>>> After adding the guest IP to the pod, still not able to add a VM >>>> instance to the cloud. Goth this popup error window: >> >>>> Unable to create a deployment for VM[User|i-2-9-VM] >> >>>> Seeing insufficient capacity for job-18 in management-server.log >>>> Attaching the log file snippet. It is hard for me to determine the >>>> precise insufficiency. Please let me know. > > >The insufficientCapacityException is the same - can't allocate Guest ip. >Note then when the first user vm is being started in the Pod, and there >is no Virtual Router running in the Pod, 2 ip addresses getting allocated: >one for the Virtual router, another for user Vm. > > > >> >>>> Also, the dashboard shows that the fourth guest ip address is still >>>> taken. When will CSMS give back the ip address of the VM instance >>>> that failed? Logging out and redisplaying the dashboard from the >>>> browser still shows the fourth direct ip is taken. Destroying all >>>> failed instances didn't help either. Clinging to that ip address >>>> seems like a bug to me. > > >The IP stays with the vm for the entire life cycle, and being released >only when vm is Expunged. That's how we ensure ip address uniqueness for >the vm for as long as vm lives. To make the ip release faster, change >parameters for expunge thread (see below). > > > > > >> >>>> Also, how long does expunging take after I destroy an instance? >>>> That is, when does the instance finally become expunged? >>>> I see instances expunging for about 24 hours now. > > > >Expunge thread runs every 24 hours; it's configurable. To make it faster, >set expunge.delay/expunge.interval global config to some small values, >say, 60 (seconds) and restart the management server process. The >Destroyed vms should be expunged shortly. > >> >>>> Regards, >>>> Evan >> >>> >>>Also, I, actually, don't see in XenCenter 6.0 what device is using the >>>end (last) guest ip address that I defined. However, it is apparently >>>used by something. What is using the last >>>(end) guest ip address? The dashboard says the 3 direct ip addresses >>>are used, but I can't identify what is using the third (last or end) >>>guest ip address. >> >> >> >>You can check cloud.nics table (select instance_id where >>ip4_address=<ip> and network_id=<id of the guest network> >> >>> >>> >>>Regards, >>>Evan >>> >>>-----Original Message----- >>>From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com] >>>Sent: Monday, June 04, 2012 9:14 PM >>>To: cloudstack-users@incubator.apache.org >>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host >>>is in avoid set) >>> >>>On 6/4/12 8:07 PM, "Evan Miller" <evan.mil...@citrix.com> wrote: >>> >>>>Attached is the proper grep of job-14. >>>> >>>>In this small size instance scenario failure, is the XenServer the >>>>system that is saying 'Hey 500Mbytes isn't enough RAM'? >>>> >>>>I am trying to understand exactly which system is saying what in >>>>particular is insufficient. >>> >>> >>> >>>Here is the reason for the failure: Nic reservation failed due to >>>Insufficient capacity exception (DirectPodBasedNetworkGuru.getIp:158): >>> >>> >>>2012-06-04 18:10:02,577 DEBUG [db.Transaction.Transaction] >>>(Job-Executor-1:job-14) Rolling back the transaction: Time = 2 Name = >>>-AsyncJobManagerImpl$1.run:393-Executors$RunnableAdapter.call:471-Futu >>>r >>>eTa >>>s >>>k$Sync.innerRun:334-FutureTask.run:166-ThreadPoolExecutor.runWorker:11 >>>1 >>>0-T >>>h >>>readPoolExecutor$Worker.run:603-Thread.run:679; called by >>>-Transaction.rollback:837-Transaction.removeUpTo:780-Transaction.close: >>>599 >>>- >>>DatabaseCallback.interceptComplete:67-DatabaseCallback.intercept:32-Ne >>>t >>>wor >>>k >>>ManagerImpl.assignPublicIpAddress:355-DirectPodBasedNetworkGuru.getIp: >>>1 >>>58- >>>D >>>irectPodBasedNetworkGuru.reserve:146-DatabaseCallback.intercept:30-Net >>>w >>>ork >>>M >>>anagerImpl.prepare:1859-VirtualMachineManagerImpl.advanceStart:741-Vir >>>t >>>ual >>>M >>>achineManagerImpl.start:461 >>>2012-06-04 18:10:02,578 INFO [cloud.vm.VirtualMachineManagerImpl] >>>(Job-Executor-1:job-14) Insufficient capacity >>> >>> >>>Looks like the Guest network doesn't have free guest IPs in the POD1 >>>targeted for deployment. To add the additional Ips, do this: >>> >>>* select Zone >>>* Select Guest Network on the network diagram >>>* Add guest IP range to the POD. >>> >>> >>>-Alena. >>> >>> >>> >>> >>>> >>>> >>>>Regards, >>>>Evan >>>> >>>>-----Original Message----- >>>>From: Alena Prokharchyk [mailto:alena.prokharc...@citrix.com] >>>>Sent: Monday, June 04, 2012 7:41 PM >>>>To: cloudstack-users@incubator.apache.org >>>>Subject: Re: Unable to add an instance - speed=500Mhz, Ram=512 (host >>>>is in avoid set) >>>> >>>>The setup lacks the resources. See more detailed information inline. >>>>Btw, to capture all logs related to vm deployment, grep by job-id >>>>should be done (in this case "grep job-14 management-server.log"). >>>> >>>>-Alena. >>>> >>>> >>>>On 6/4/12 7:27 PM, "Evan Miller" <evan.mil...@citrix.com> wrote: >>>> >>>>>Trying to setup a single, small VM to my cloud. >>>>>My cloud, so far, consists of: >>>>> >>>>>CloudStack Management Server: >>>>> * OS = Centos 6.2 x86 64-bit >>>>> * CPU = Intel 3050, 2.13 GHz >>>>> * RAM = 2 Gbytes >>>>> >>>>>XenServer: >>>>> * OS = Provided by XenServer 6.0.2 CD >>>>> * Hardware = IBM 3250 >>>>> * CPU = Intel Xeon Dual Core, Single Physical CPU@ 1.86 GHz >>>>> >>>>>Template: >>>>>* CentOS 5.6(64-bit) no GUI (XenServer) >>>>> >>>>>Per the management log below, is XenServer refusing to setup the >>>>>first VM? >>>>> >>>>> >>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) DeploymentPlanner allocation algorithm: >>>>>random >>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Trying to allocate a host and storage pools >>>>>from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: >>>>>536870912 >>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Is ROOT volume READY (pool already >>>>>allocated)?: >>>>>No >>>>>2012-06-04 18:10:02,907 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Searching all possible resources under this >>>>>Zone: >>>>>1 >>>>>2012-06-04 18:10:02,909 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Listing clusters in order of aggregate >>>>>capacity, that have (atleast one host with) enough CPU and RAM >>>>>capacity under this >>>>>Zone: 1 >>>>>2012-06-04 18:10:02,910 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) CPUOverprovisioningFactor considered: 1.0 >>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Checking resources in Cluster: 1 under Pod: >>>>>1 >>>>>2012-06-04 18:10:02,918 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Calling HostAllocators to find suitable >>>>>hosts >>>>>2012-06-04 18:10:02,918 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for hosts >>>>>in >>>>>dc: >>>>>1 pod:1 cluster:1 >>>>>2012-06-04 18:10:02,920 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) FirstFitAllocator >>>>>has >>>>>1 hosts to check for allocation: [Host[-1-Routing]] >>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Found 1 hosts for >>>>>allocation after prioritization: [Host[-1-Routing]] >>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Looking for >>>>>speed=500Mhz, Ram=512 >>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host name: >>>>>xenserver1-cs, hostId: 1 is in avoid set, skipping this and trying >>>>>other available hosts >>>> >>>> >>>>The host was put in avoid set some time before (the log fragment >>>>you've sent doesn't capture it). The host can be put in avoid set >>>>when it doesn't have enough resources for vm start (cpu/ram). >>>> >>>> >>>> >>>>>2012-06-04 18:10:02,923 DEBUG [allocator.impl.FirstFitAllocator] >>>>>(Job-Executor-1:job-14 FirstFitRoutingAllocator) Host Allocator >>>>>returning 0 suitable hosts >>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) No suitable hosts found >>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) No suitable hosts found under this Cluster: >>>>>1 >>>>>2012-06-04 18:10:02,923 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-1:job-14) Could not find suitable Deployment >>>>>Destination for this VM under any clusters, returning. >>>>>2012-06-04 18:10:03,022 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-1:job-14) VM state transitted from :Starting to >>>>>Stopped with event: OperationFailedvm's original host id: null new >>>>>host id: >>>>>null host id before state transition: 1 >>>>>2012-06-04 18:10:03,027 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-1:job-14) Hosts's actual total CPU: 3732 and CPU after >>>>>applying overprovisioning: 3732 >>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-1:job-14) release cpu from host: 1, old used: >>>>>1500,reserved: 0, actual total: 3732, total with overprovisioning: >>>>>3732; new used: 1000,reserved:0; movedfromreserved: >>>>>false,moveToReserveredfalse >>>>>2012-06-04 18:10:03,028 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-1:job-14) release mem from host: 1, old used: >>>>>1879048192,reserved: 0, total: 3775173696; new used: >>>>>1342177280,reserved:0; movedfromreserved: >>>>>false,moveToReserveredfalse >>>>>2012-06-04 18:10:03,183 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-1:job-14) VM state transitted from :Stopped to Error >>>>>with >>>>>event: OperationFailedToErrorvm's original host id: null new host id: >>>>>null host id before state transition: null >>>>>2012-06-04 18:10:03,888 INFO [api.commands.DeployVMCmd] >>>>>(Job-Executor-1:job-14) >>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to >>>>>create a deployment for VM[User|i-2-7-VM]Scope=interface >>>>>com.cloud.dc.DataCenter; id=1 >>>>>2012-06-04 18:10:03,888 WARN [cloud.api.ApiDispatcher] >>>>>(Job-Executor-1:job-14) class com.cloud.api.ServerApiException : >>>>>Unable to create a deployment for VM[User|i-2-7-VM] >>>>>2012-06-04 18:10:03,891 DEBUG [cloud.async.AsyncJobManagerImpl] >>>>>(Job-Executor-1:job-14) Complete async job-14, jobStatus: 2, >>>>>resultCode: >>>>>530, result: >>>>>com.cloud.api.response.ExceptionResponse@4ebe0382<mailto:com.cloud.api >>>>>. >>>>>res >>>>>ponse.ExceptionResponse@4ebe0382> >>>>>2012-06-04 18:10:07,258 DEBUG [cloud.async.AsyncJobManagerImpl] >>>>>(catalina-exec-7:null) Async job-14 completed >>>>> >>>>> >>>>> >>>>>I, also, tried to create a medium size VM. This, also, failed with a >>>>>different error related to InsufficientServerCapacityException: >>>>> >>>>> >>>>> >>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) DeploymentPlanner allocation algorithm: >>>>>random >>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Trying to allocate a host and storage pools >>>>>from dc:1, pod:null,cluster:null, requested cpu: 1000, requested ram: >>>>>1073741824 >>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Is ROOT volume READY (pool already allocated)?: >>>>>No >>>>>2012-06-04 17:34:52,550 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Searching all possible resources under this >>>>>Zone: 1 >>>>>2012-06-04 17:34:52,551 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Listing clusters in order of aggregate >>>>>capacity, that have (atleast one host with) enough CPU and RAM >>>>>capacity under this >>>>>Zone: 1 >>>>>2012-06-04 17:34:52,552 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) CPUOverprovisioningFactor considered: 1.0 >>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Cannot allocate cluster 1 for vm creation >>>>>since its allocated percentage: 0.9243709585329766 will cross the >>>>>disable capacity threshold: 0.85 for capacity Type : 0, skipping >>>>>this cluster >>>> >>>> >>>>CapacityType: 0 means RAM. When we analyze if host is capable for vm >>>>deployment, we always make sure the total RAM (including the RAM we >>>>are allocating for the vm), doesn't exceed RAM capacity threshold >>>>(global config variable >>>>cluster.memory.allocated.capacity.disablethreshold). >>>> >>>> >>>> >>>>>2012-06-04 17:34:52,558 DEBUG [cloud.deploy.FirstFitPlanner] >>>>>(Job-Executor-9:job-9) Could not find suitable Deployment >>>>>Destination for this VM under any clusters, returning. >>>>>2012-06-04 17:34:52,690 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-9:job-9) VM state transitted from :Starting to Stopped >>>>>with >>>>>event: OperationFailedvm's original host id: null new host id: null >>>>>host id before state transition: 1 >>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-9:job-9) Hosts's actual total CPU: 3732 and CPU after >>>>>applying overprovisioning: 3732 >>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-9:job-9) release cpu from host: 1, old used: >>>>>2000,reserved: >>>>>0, actual total: 3732, total with overprovisioning: 3732; new used: >>>>>1000,reserved:0; movedfromreserved: false,moveToReserveredfalse >>>>>2012-06-04 17:34:52,694 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-9:job-9) release mem from host: 1, old used: >>>>>2415919104,reserved: 0, total: 3775173696; new used: >>>>>1342177280,reserved:0; movedfromreserved: >>>>>false,moveToReserveredfalse >>>>>2012-06-04 17:34:52,842 DEBUG [cloud.capacity.CapacityManagerImpl] >>>>>(Job-Executor-9:job-9) VM state transitted from :Stopped to Error >>>>>with >>>>>event: OperationFailedToErrorvm's original host id: null new host id: >>>>>null host id before state transition: null >>>>>2012-06-04 17:34:53,463 INFO [api.commands.DeployVMCmd] >>>>>(Job-Executor-9:job-9) >>>>>com.cloud.exception.InsufficientServerCapacityException: Unable to >>>>>create a deployment for VM[User|i-2-5-VM]Scope=interface >>>>>com.cloud.dc.DataCenter; id=1 >>>>>2012-06-04 17:34:53,464 WARN [cloud.api.ApiDispatcher] >>>>>(Job-Executor-9:job-9) class com.cloud.api.ServerApiException : >>>>>Unable to create a deployment for VM[User|i-2-5-VM] >>>>>2012-06-04 17:34:53,464 DEBUG [cloud.async.AsyncJobManagerImpl] >>>>>(Job-Executor-9:job-9) Complete async job-9, jobStatus: 2, resultCode: >>>>>530, result: >>>>>com.cloud.api.response.ExceptionResponse@7f378496<mailto:com.cloud.api >>>>>. >>>>>res >>>>>ponse.ExceptionResponse@7f378496> >>>>>2012-06-04 17:34:56,985 DEBUG [cloud.async.AsyncJobManagerImpl] >>>>>(catalina-exec-17:null) Async job-9 completed >>>>> >>>>>How to overcome these errors? >>>> >>>> >>>>The solution will be - create the service offering with small amount >>>>of RAM/CPU (depends of how many vms you are planning to run on your >>>>host). >>>>Or add more hosts to the Xen cluster. >>>> >>>>> >>>>>Regards, >>>>> >>>>>Evan Miller >>>>> >>>>> >>>> >>>> >>>> >>> >>> >>> >> >> >> > > >
