Hi, I am able to setup succesfully the cloudstack and created the instances running successfully, i am able to stop and start the instances succesfully.
but suddenly after stopping and starting the instance it gives me following exception... may i know what is the reason when i am checking the logs it shows like host is avoid set, pod is avoid set... and finally throws the insufficent server capacity exception. May i know what is the avoid set how to overcome this issue... if it is the insufficient server capacity issue it should throw this exception long back. but it is started throwing exception today onwards... iam confused and stuck pls help me on this below is the logs pasted 2013-12-30 10:52:37,434 DEBUG [agent.manager. AgentManagerImpl] (AgentManager-Handler-14:null) SeqA 2-24333: Sending Seq 2-24333: { Ans: , MgmtId: 90591532766, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-12-30 10:52:39,290 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector is running... 2013-12-30 10:52:39,811 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380873338 2013-12-30 10:52:39,889 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380873338 2013-12-30 10:52:42,811 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===START=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380876338 2013-12-30 10:52:42,882 DEBUG [cloud.api.ApiServlet] (catalina-exec-13:null) ===END=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380876338 2013-12-30 10:52:43,115 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-1087657629: Received: { Ans: , MgmtId: 90591532766, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2013-12-30 10:52:45,814 DEBUG [cloud.api.ApiServlet] (catalina-exec-12:null) ===START=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380879340 2013-12-30 10:52:45,883 DEBUG [cloud.api.ApiServlet] (catalina-exec-12:null) ===END=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380879340 2013-12-30 10:52:47,358 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 2-24334: Processing Seq 2-24334: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-12-30 10:52:47,424 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 2-24334: Sending Seq 2-24334: { Ans: , MgmtId: 90591532766, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-12-30 10:52:48,817 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380882337 2013-12-30 10:52:48,903 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===END=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380882337 2013-12-30 10:52:49,115 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector is running... 2013-12-30 10:52:51,312 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-1087657630: Received: { Ans: , MgmtId: 90591532766, via: 1, Ver: v1, Flags: 10, { GetHostStatsAnswer } } 2013-12-30 10:52:51,673 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM 2013-12-30 10:52:51,811 DEBUG [cloud.api.ApiServlet] (catalina-exec-9:null) ===START=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380885337 2013-12-30 10:52:51,889 DEBUG [cloud.api.ApiServlet] (catalina-exec-9:null) ===END=== 10.203.251.143 -- GET command=queryAsyncJobResult&jobId=4333d42a-03b4-4500-97db-429e744df78b&response=json&sessionkey=7JJVl1vgRpS%2FxVXGO1y4KyXksYo%3D&_=1388380885337 2013-12-30 10:52:51,910 DEBUG [agent.manager.AgentAttache] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Seq 1-1087657367: Waiting some more time because this is the current command 2013-12-30 10:52:51,966 DEBUG [agent.transport.Request] (AgentManager-Handler-10:null) Seq 1-1087657367: Processing: { Ans: , MgmtId: 90591532766, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.StopAnswer":{"vncPort":0,"result":true,"wait":0}}] } 2013-12-30 10:52:51,966 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-10:null) Seq 1-1087657367: No more commands found 2013-12-30 10:52:51,966 DEBUG [agent.transport.Request] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Seq 1-1087657367: Received: { Ans: , MgmtId: 90591532766, via: 1, Ver: v1, Flags: 110, { StopAnswer } } 2013-12-30 10:52:51,973 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Changing active number of nics for network id=204 on -1 2013-12-30 10:52:52,139 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy 2013-12-30 10:52:52,143 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Asking VirtualRouter to release Nic[8-3-c3ad5028-1525-4ae5-9b1b-38d1012465bd-10.203.251.170] 2013-12-30 10:52:52,145 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Asking SecurityGroupProvider to release Nic[8-3-c3ad5028-1525-4ae5-9b1b-38d1012465bd-10.203.251.170] 2013-12-30 10:52:52,145 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Successfully released network resources for the vm VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d] 2013-12-30 10:52:52,145 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Successfully cleanued up resources for the vm VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d] in Starting state 2013-12-30 10:52:52,146 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Root volume is ready, need to place VM in volume's cluster 2013-12-30 10:52:52,150 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Deploy avoids pods: [1], clusters: [], hosts: [1] 2013-12-30 10:52:52,151 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_f14643aa@672774c8 2013-12-30 10:52:52,151 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:1, requested cpu: 1000, requested ram: 1073741824 2013-12-30 10:52:52,151 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Is ROOT volume READY (pool already allocated)?: Yes 2013-12-30 10:52:52,151 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) DeploymentPlan has host_id specified, choosing this host and making no checks on this host: 1 2013-12-30 10:52:52,152 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) The specified host is in avoid set 2013-12-30 10:52:52,152 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Cannnot deploy to specified host, returning. 2013-12-30 10:52:52,360 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) SeqA 2-24335: Processing Seq 2-24335: { Cmd , MgmtId: -1, via: 2, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2013-12-30 10:52:52,380 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: 1 2013-12-30 10:52:52,383 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Hosts's actual total CPU: 4666 and CPU after applying overprovisioning: 4666 2013-12-30 10:52:52,383 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Hosts's actual total RAM: 4016525312 and RAM after applying overprovisioning: 4016525312 2013-12-30 10:52:52,383 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) release cpu from host: 1, old used: 2500,reserved: 0, actual total: 4666, total with overprovisioning: 4666; new used: 1500,reserved:0; movedfromreserved: false,moveToReserveredfalse 2013-12-30 10:52:52,383 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) release mem from host: 1, old used: 2550136832,reserved: 0, total: 4016525312; new used: 1476395008,reserved:0; movedfromreserved: false,moveToReserveredfalse 2013-12-30 10:52:52,431 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-5:null) SeqA 2-24335: Sending Seq 2-24335: { Ans: , MgmtId: 90591532766, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2013-12-30 10:52:52,585 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: 1 new host id: null host id before state transition: null 2013-12-30 10:52:52,585 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Successfully transitioned to start state for VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d] reservation id = d4264a85-92c6-4a7e-aeff-fca97e2ac5ca 2013-12-30 10:52:52,633 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Trying to deploy VM, vm has dcId: 1 and podId: 1 2013-12-30 10:52:52,633 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Deploy avoids pods: [1], clusters: [], hosts: [1] 2013-12-30 10:52:52,634 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Root volume is ready, need to place VM in volume's cluster 2013-12-30 10:52:52,634 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Vol[3|vm=3|ROOT] is READY, changing deployment plan to use this pool's dcId: 1 , podId: 1 , and clusterId: 1 2013-12-30 10:52:52,639 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Deploy avoids pods: [1], clusters: [], hosts: [1] 2013-12-30 10:52:52,639 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_f14643aa@672774c8 2013-12-30 10:52:52,639 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:1, requested cpu: 1000, requested ram: 1073741824 2013-12-30 10:52:52,639 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Is ROOT volume READY (pool already allocated)?: Yes 2013-12-30 10:52:52,639 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) This VM has last host_id specified, trying to choose the same host: 1 2013-12-30 10:52:52,640 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) The last host of this VM is in avoid set 2013-12-30 10:52:52,640 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Cannot choose the last host to deploy this VM 2013-12-30 10:52:52,640 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Searching resources only under specified Cluster: 1 2013-12-30 10:52:52,641 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) The specified cluster is in avoid set, returning. 2013-12-30 10:52:52,646 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Deploy avoids pods: [1], clusters: [], hosts: [1] 2013-12-30 10:52:52,646 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_f14643aa@672774c8 2013-12-30 10:52:52,646 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Trying to allocate a host and storage pools from dc:1, pod:1,cluster:null, requested cpu: 1000, requested ram: 1073741824 2013-12-30 10:52:52,646 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Is ROOT volume READY (pool already allocated)?: No 2013-12-30 10:52:52,646 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) This VM has last host_id specified, trying to choose the same host: 1 2013-12-30 10:52:52,647 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) The last host of this VM is in avoid set 2013-12-30 10:52:52,647 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Cannot choose the last host to deploy this VM 2013-12-30 10:52:52,647 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Searching resources only under specified Pod: 1 2013-12-30 10:52:52,647 DEBUG [cloud.deploy.FirstFitPlanner] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) The specified pod is in avoid set, returning. 2013-12-30 10:52:52,749 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: 1 new host id: null host id before state transition: null 2013-12-30 10:52:52,875 INFO [user.vm.StartVMCmd] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d]Scope=interface com.cloud.dc.DataCenter; id=1 2013-12-30 10:52:52,875 INFO [user.vm.StartVMCmd] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Unable to create a deployment for VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d] com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d]Scope=interface com.cloud.dc.DataCenter; id=1 at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:842) at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578) at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237) at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3406) at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:1948) at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) at org.apache.cloudstack.api.command.user.vm.StartVMCmd.execute(StartVMCmd.java:120) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158) at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) 2013-12-30 10:52:52,876 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-12:job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ]) Complete async job-68 = [ 4333d42a-03b4-4500-97db-429e744df78b ], jobStatus: 2, resultCode: 530, result: Error Code: 533 Error text: Unable to create a deployment for VM[User|aeb8dd15-3bae-4543-aa88-e8934cfbf90d] 2013-12-30 10:52:53,579 DEBUG [cloud.network.ExternalDeviceUsageManagerImpl] (ExternalNetworkMonitor-1:null) External devices stats collector is running... 2013-12-30 10:52:53,901 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-2:null) checking health of usage server -- Regards R D Sabareesh 8050414378