Here is another part of log e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489660 to Host 3 timed out after 3600 2014-03-19 13:59:22,969 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi medoutException: Commands 1710489660 to Host 3 timed out after 3600 2014-03-19 13:59:58,120 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 13:59:58,120 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744901: Timed out on null 2014-03-19 13:59:58,120 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698 744901 to Host 4 timed out after 3600 2014-03-19 13:59:58,122 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 stati stics. 2014-03-19 13:59:58,122 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host : 4 2014-03-19 14:00:07,507 WARN [apache.cloudstack.alerts] (HA-2:null) alertType:: 13 // dataCenterId:: 0 // podId:: 0 // clusterId:: null // message:: No usage server process running 2014-03-19 14:00:22,973 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:00:22,973 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489661: Timed out on null 2014-03-19 14:00:22,973 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-1:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489661 to Host 3 timed out after 3600 2014-03-19 14:00:22,973 ERROR [cloud.server.StatsCollector] (StatsCollector-1:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTi medoutException: Commands 1710489661 to Host 3 timed out after 3600 2014-03-19 14:00:58,715 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:00:58,715 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744902: Timed out on null 2014-03-19 14:00:58,716 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698 744902 to Host 4 timed out after 3600 2014-03-19 14:00:58,716 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 stati stics. 2014-03-19 14:00:58,716 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host : 4 2014-03-19 14:01:22,978 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: c om.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:01:22,978 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 3-1710489662: Timed out on null 2014-03-19 14:01:22,978 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-2:null) Failed to send command, du e to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600 2014-03-19 14:01:22,978 ERROR [cloud.server.StatsCollector] (StatsCollector-2:null) Error trying to retrieve storage sta ts com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489662 to Host 3 timed out after 3600 2014-03-19 14:01:35,900 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3660 2014-03-19 14:01:35,900 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-1 = [ d2584efe-01e9-42bf-a5a1-e9871265e5b3 ]) Failed to start instance VM[User|Ubuntu-mysql] com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM due to concurrent operation Caused by: com.cloud.exception.ConcurrentOperationException: There are concurrent operations on VM[DomainRouter|r-4-VM] 2014-03-19 14:01:59,312 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:01:59,312 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744904: Timed out on null 2014-03-19 14:01:59,312 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744904 to Host 4 timed out after 3600 2014-03-19 14:01:59,312 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:01:59,312 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:02:07,765 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720 2014-03-19 14:02:07,766 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Failed to deploy vm 4 with original planner, sending HAPlanner 2014-03-19 14:02:07,768 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Unable to transition into Starting state due to Unable to transition to a new state from Starting via StartRequested 2014-03-19 14:02:07,811 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) Determining why we're unable to update the state to Starting for VM[DomainRouter|r-4-VM]. Retry=4 2014-03-19 14:02:07,812 WARN [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-0:work-5) The task item for vm VM[DomainRouter|r-4-VM] has been inactive for 3720 2014-03-19 14:02:07,812 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-5) Unable to restart VM[DomainRouter|r-4-VM] due to There are concurrent operations on VM[DomainRouter|r-4-VM] 2014-03-19 14:02:07,812 WARN [apache.cloudstack.alerts] (HA-Worker-0:work-5) alertType:: 9 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to restart r-4-VM which was running on host name: server2(id:1), availability zone: zone1, pod: pod1 2014-03-19 14:02:22,983 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:02:22,983 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489663: Timed out on null 2014-03-19 14:02:22,983 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600 2014-03-19 14:02:22,983 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489663 to Host 3 timed out after 3600 2014-03-19 14:02:59,904 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:02:59,904 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 4-698744905: Timed out on null 2014-03-19 14:02:59,905 WARN [agent.manager.AgentManagerImpl] (StatsCollector-3:null) Operation timed out: Commands 698 744905 to Host 4 timed out after 3600 2014-03-19 14:02:59,905 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-3:null) Unable to obtain host 4 statistics. 2014-03-19 14:02:59,905 WARN [cloud.server.StatsCollector] (StatsCollector-3:null) Received invalid host stats for host: 4 2014-03-19 14:03:22,988 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:03:22,988 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489664: Timed out on null 2014-03-19 14:03:22,988 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600 2014-03-19 14:03:22,988 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489664 to Host 3 timed out after 3600 2014-03-19 14:04:00,500 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:04:00,500 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744906: Timed out on null 2014-03-19 14:04:00,501 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744906 to Host 4 timed out after 3600 2014-03-19 14:04:00,501 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:04:00,501 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:04:22,993 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:04:22,993 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489665: Timed out on null 2014-03-19 14:04:22,993 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600 2014-03-19 14:04:22,993 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489665 to Host 3 timed out after 3600 2014-03-19 14:05:01,092 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-2:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:05:01,092 WARN [agent.manager.AgentAttache] (StatsCollector-2:null) Seq 4-698744907: Timed out on null 2014-03-19 14:05:01,092 WARN [agent.manager.AgentManagerImpl] (StatsCollector-2:null) Operation timed out: Commands 698744907 to Host 4 timed out after 3600 2014-03-19 14:05:01,092 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-2:null) Unable to obtain host 4 statistics. 2014-03-19 14:05:01,092 WARN [cloud.server.StatsCollector] (StatsCollector-2:null) Received invalid host stats for host : 4 2014-03-19 14:05:23,000 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-3:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:05:23,000 WARN [agent.manager.AgentAttache] (StatsCollector-3:null) Seq 3-1710489666: Timed out on null 2014-03-19 14:05:23,000 DEBUG [cloudstack.storage.RemoteHostEndPoint] (StatsCollector-3:null) Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600 2014-03-19 14:05:23,000 ERROR [cloud.server.StatsCollector] (StatsCollector-3:null) Error trying to retrieve storage stats com.cloud.utils.exception.CloudRuntimeException: Failed to send command, due to Agent:3, com.cloud.exception.OperationTimedoutException: Commands 1710489666 to Host 3 timed out after 3600 2014-03-19 14:06:01,680 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:06:01,680 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 4-698744908: Timed out on null 2014-03-19 14:06:01,680 WARN [agent.manager.AgentManagerImpl] (StatsCollector-1:null) Operation timed out: Commands 698744908 to Host 4 timed out after 3600 2014-03-19 14:06:01,680 WARN [cloud.resource.ResourceManagerImpl] (StatsCollector-1:null) Unable to obtain host 4 statistics. 2014-03-19 14:06:01,681 WARN [cloud.server.StatsCollector] (StatsCollector-1:null) Received invalid host stats for host: 4 2014-03-19 14:06:23,004 INFO [utils.exception.CSExceptionErrorCode] (StatsCollector-1:null) Could not find exception: com.cloud.exception.OperationTimedoutException in error code list for exceptions 2014-03-19 14:06:23,004 WARN [agent.manager.AgentAttache] (StatsCollector-1:null) Seq 3-1710489667: Timed out on null 2014-03-19 14:06:23,004 DEB
On Wednesday, 19 March 2014 2:36 PM, Suresh Sadhu <suresh.sa...@citrix.com> wrote: Can you please provide the logs and also did you notice any exception in the management log. For deleting vm : You can update the vm state in db as Stopped and try to delete them from CS. Regards Sadhu -----Original Message----- From: Sugandh S [mailto:s.suga...@rocketmail.com] Sent: 19 March 2014 14:30 To: users@cloudstack.apache.org Subject: vm stuck in starting state, unable to delete it Hello all, I am using CS 4.2 and my setup is as follows: One server, running Ubuntu 12.04, is serving as both Cloudstack-management server and Cloudstack-agent. Primary storage and secondary storage are also provided by this server via NFS. For primary storage, export location is /export/primary and for secondary storage, it is /export/secondary. Second server, also running Ubuntu 12.04, only serves as Cloudstack-agent. Now, when I create vms they are stuck in starting state and I am unable to delete them. Any and all help would be greatly appreciated. Thanks ahead, Sugandh
