Hi, I am having a problem with the ConsoleProxy VM starting up, in the CS UI the consoleproxy is showing as "Starting up" and remains in this state. I have restarted the cloud-management service as well as the console VM on the host, but still appears to be hung.
I am unable to restart or destroy the console VM in the UI, as no such option. Any help would be appreciated. The CS management log shows the following: 2013-04-01 21:35:35,553 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking ConsoleProxyAlert to shutdown. 2013-04-01 21:35:36,626 INFO [utils.component.ComponentLocator] (Thread-5:null) Asking Manager ConsoleProxyManager to shutdown. 2013-04-01 21:35:36,626 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (Thread-5:null) Stop console proxy manager 2013-04-01 21:35:37,626 INFO [utils.component.ComponentLocator] (Thread-5:null) Skipping Manager ConsoleProxyManager because it is not in a state to shutdown. 2013-04-01 21:35:47,256 DEBUG [utils.component.ComponentLocator] (main:null) Looking for class com.cloud.consoleproxy.ConsoleProxyBalanceAllocator 2013-04-01 21:35:47,257 INFO [utils.component.ComponentLocator] (main:null) Found component: com.cloud.consoleproxy.ConsoleProxyAllocator in com.cloud.consoleproxy.ConsoleProxyBalanceAllocator - Balance 2013-04-01 21:35:47,333 DEBUG [utils.component.ComponentLocator] (main:null) Looking for class com.cloud.alert.ConsoleProxyAlertAdapter 2013-04-01 21:35:47,334 INFO [utils.component.ComponentLocator] (main:null) Found component: com.cloud.alert.AlertAdapter in com.cloud.alert.ConsoleProxyAlertAdapter - ConsoleProxyAlert 2013-04-01 21:35:47,518 INFO [utils.component.ComponentLocator] (main:null) Found component: com.cloud.vm.dao.ConsoleProxyDao in com.cloud.vm.dao.ConsoleProxyDaoImpl - ConsoleProxyDao 2013-04-01 21:35:47,697 INFO [utils.component.ComponentLocator] (main:null) Found component: com.cloud.consoleproxy.ConsoleProxyManager in com.cloud.consoleproxy.ConsoleProxyManagerImpl - ConsoleProxyManager 2013-04-01 21:35:47,697 INFO [utils.component.ComponentLocator] (main:null) Found component: com.cloud.consoleproxy.ConsoleProxyService in com.cloud.consoleproxy.ConsoleProxyManagerImpl - ConsoleProxyManager 2013-04-01 21:35:50,355 INFO [utils.component.ComponentLocator] (main:null) Starting singleton DAO: ConsoleProxyDao 2013-04-01 21:35:52,284 INFO [utils.component.ComponentLocator] (main:null) Instantiating Adapter: ConsoleProxyAlert 2013-04-01 21:35:52,682 INFO [utils.component.ComponentLocator] (main:null) Instantiating Manager: ConsoleProxyManager 2013-04-01 21:35:52,937 DEBUG [utils.component.ComponentLocator] (main:null) Injecting singleton Manager: ConsoleProxyManager 2013-04-01 21:35:53,364 INFO [utils.component.ComponentLocator] (main:null) Configuring singleton Manager: ConsoleProxyManager 2013-04-01 21:35:53,364 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (main:null) Start configuring console proxy manager : ConsoleProxyManager 2013-04-01 21:35:53,364 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (main:null) Console proxy max session soft limit : 50 2013-04-01 21:35:53,364 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (main:null) Console proxy standby capacity : 10 2013-04-01 21:35:53,380 DEBUG [agent.manager.AgentManagerImpl] (main:null) Registering listener ConsoleProxyListener with id 12 2013-04-01 21:35:53,383 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (main:null) Console Proxy Manager is configured. 2013-04-01 21:35:53,511 INFO [utils.component.ComponentLocator] (main:null) Injecting singleton Adapter: ConsoleProxyAlert 2013-04-01 21:35:53,511 INFO [utils.component.ComponentLocator] (main:null) Configuring singleton Adapter: ConsoleProxyAlert 2013-04-01 21:35:53,511 INFO [cloud.alert.ConsoleProxyAlertAdapter] (main:null) Start configuring console proxy alert manager : ConsoleProxyAlert 2013-04-01 21:35:53,838 INFO [cloud.ha.HighAvailabilityManagerImpl] (main:null) Schedule vm for HA: VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:53,880 INFO [cloud.ha.HighAvailabilityManagerImpl] (main:null) Schedule vm for HA: VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:53,886 INFO [utils.component.ComponentLocator] (main:null) Starting singleton Manager: ConsoleProxyManager 2013-04-01 21:35:53,886 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (main:null) Start console proxy manager 2013-04-01 21:35:53,886 INFO [utils.component.ComponentLocator] (main:null) Started Manager: ConsoleProxyManager 2013-04-01 21:35:54,004 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) HA on VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:54,014 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) HA on VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:54,030 INFO [utils.component.ComponentLocator] (main:null) Starting singleton Adapter: ConsoleProxyAlert 2013-04-01 21:35:54,065 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-1:work-38) Unable to reach the agent for VM[ConsoleProxy|v-2-VM]: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected 2013-04-01 21:35:54,066 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) SimpleInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,066 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) VmwareInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,075 DEBUG [cloud.ha.CheckOnAgentInvestigator] (HA-Worker-0:work-37) Unable to reach the agent for VM[ConsoleProxy|v-2-VM]: Resource [Host:1] is unreachable: Host 1: Host with specified id is not in the right state: Disconnected 2013-04-01 21:35:54,079 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) SimpleInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,079 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) VmwareInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,084 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) XenServerInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,084 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-0:work-37) Not a User Vm, unable to determine state of VM[ConsoleProxy|v-2-VM] returning null 2013-04-01 21:35:54,084 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) PingInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,084 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-37) Testing if VM[ConsoleProxy|v-2-VM] is alive 2013-04-01 21:35:54,090 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) XenServerInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,090 DEBUG [cloud.ha.UserVmDomRInvestigator] (HA-Worker-1:work-38) Not a User Vm, unable to determine state of VM[ConsoleProxy|v-2-VM] returning null 2013-04-01 21:35:54,094 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) PingInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,094 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-1:work-38) Testing if VM[ConsoleProxy|v-2-VM] is alive 2013-04-01 21:35:54,123 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-1:work-38) unable to determine state of VM[ConsoleProxy|v-2-VM] returning null 2013-04-01 21:35:54,123 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) ManagementIPSysVMInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,124 DEBUG [cloud.ha.ManagementIPSystemVMInvestigator] (HA-Worker-0:work-37) unable to determine state of VM[ConsoleProxy|v-2-VM] returning null 2013-04-01 21:35:54,124 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) ManagementIPSysVMInvestigator found VM[ConsoleProxy|v-2-VM]to be alive? null 2013-04-01 21:35:54,127 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-1:work-38) Unable to fence off VM[ConsoleProxy|v-2-VM] on Host[-1-Routing] 2013-04-01 21:35:54,127 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-38) We were unable to fence off the VM VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:54,150 DEBUG [cloud.ha.XenServerFencer] (HA-Worker-0:work-37) Unable to fence off VM[ConsoleProxy|v-2-VM] on Host[-1-Routing] 2013-04-01 21:35:54,163 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-0:work-37) We were unable to fence off the VM VM[ConsoleProxy|v-2-VM] 2013-04-01 21:35:57,208 DEBUG [cloud.resource.ResourceManagerImpl] (AgentConnectTaskPool-1:null) Dispatching resource state event CREATE_HOST_VO_FOR_CONNECTED to ConsoleProxyManagerImpl$$EnhancerByCGLIB$$bdec6fa 2013-04-01 21:36:06,270 DEBUG [agent.manager.AgentManagerImpl] (AgentConnectTaskPool-1:null) Sending Connect to listener: ConsoleProxyListener 2013-04-01 21:36:08,058 DEBUG [cloud.resource.ResourceManagerImpl] (AgentTaskPool-1:null) Dispatching resource state event CREATE_HOST_VO_FOR_DIRECT_CONNECT to ConsoleProxyManagerImpl$$EnhancerByCGLIB$$bdec6fa 2013-04-01 21:36:10,577 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-1:null) Found an outstanding work item for this vm VM[ConsoleProxy|v-2-VM] in state:Starting, work id:d1de55ad-2517-466a-9b23-9d3caee2c9e9 2013-04-01 21:36:10,625 DEBUG [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) VM does not require investigation so I'm marking it as Stopped: VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:31,738 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-1:null) VM[ConsoleProxy|v-2-VM] is stopped on the host. Proceeding to release resource held. 2013-04-01 21:36:31,809 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-1:null) Successfully released network resources for the vm VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:31,809 DEBUG [cloud.vm.VirtualMachineManagerImpl] (AgentTaskPool-1:null) Successfully released storage resources for the vm VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:31,855 INFO [cloud.ha.HighAvailabilityManagerImpl] (AgentTaskPool-1:null) Schedule vm for HA: VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:31,881 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-39) HA on VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:32,613 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-1:null) Sending Connect to listener: ConsoleProxyListener 2013-04-01 21:36:53,413 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy 2013-04-01 21:36:53,413 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Expand console proxy standby capacity for zone London 2013-04-01 21:36:53,418 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Found a stopped console proxy, bring it up to running pool. proxy vm id : 2 2013-04-01 21:36:53,442 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Successfully transitioned to start state for VM[ConsoleProxy|v-2-VM] reservation id = e060dfdb-73b7-4b3c-aef3-86e86fb41ea2 2013-04-01 21:36:53,505 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Deployment found - P0=VM[ConsoleProxy|v-2-VM], P0=Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod(1)-Cluster(1)-Host(1)-Storage()] 2013-04-01 21:36:53,744 DEBUG [network.guru.PodBasedNetworkGuru] (consoleproxy-1:null) Allocated a nic NicProfile[7-2-e060dfdb-73b7-4b3c-aef3-86e86fb41ea2-192.168.2.94-null for VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:53,752 DEBUG [cloud.storage.StorageManagerImpl] (consoleproxy-1:null) Checking if we need to prepare 1 volumes for VM[ConsoleProxy|v-2-VM] 2013-04-01 21:36:53,812 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Boot Args for VM[ConsoleProxy|v-2-VM]: template=domP type=consoleproxy host=192.168.2.201 port=8250 name=v-2-VM premium=true zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true eth2ip=192.168.2.125 eth2mask=255.255.255.0 gateway=192.168.2.1 eth0ip=169.254.1.78 eth0mask=255.255.0.0 eth1ip=192.168.2.94 eth1mask=255.255.255.0 mgmtcidr=192.168.2.0/24 localgw=192.168.2.1 internaldns1=192.168.2.1 dns1=8.8.8.8 dns2=8.8.4.4 2013-04-01 21:36:53,854 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 1-557383690: Sending { Cmd , MgmtId: 52232380435, via: 1, Ver: v1, Flags: 100111, [{"StartCommand":{"vm":{"id":2,"name":"v-2-VM","bootloader":"PyGrub","type":"ConsoleProxy","cpus":1,"speed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 6(64-bit)","bootArgs":" template=domP type=consoleproxy host=192.168.2.201 port=8250 name=v-2-VM premium=true zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true eth2ip=192.168.2.125 eth2mask=255.255.255.0 gateway=192.168.2.1 eth0ip=169.254.1.78 eth0mask=255.255.0.0 eth1ip=192.168.2.94 eth1mask=255.255.255.0 mgmtcidr= 192.168.2.0/24 localgw=192.168.2.1 internaldns1=192.168.2.1 dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword":"38be171d61aa8901","params":{},"disks":[{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"58945947-d40e-4532-b26a-f97f696715f8","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"c06d70ac-1552-3d7d-afe7-07d5690d2e09","deviceId":0}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"ip":"192.168.2.125","netmask":"255.255.255.0","gateway":"192.168.2.1","mac":"06:3a:44:00:00:2e","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"ip":"169.254.1.78","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:4e","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"ip":"192.168.2.94","netmask":"255.255.255.0","gateway":"192.168.2.1","mac":"06:51:7e:00:00:0f","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false}]},"wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.78","port":3922,"interval":6,"retries":100,"name":"v-2-VM","wait":0}}] } 2013-04-01 21:36:53,859 DEBUG [agent.transport.Request] (consoleproxy-1:null) Seq 1-557383690: Executing: { Cmd , MgmtId: 52232380435, via: 1, Ver: v1, Flags: 100111, [{"StartCommand":{"vm":{"id":2,"name":"v-2-VM","bootloader":"PyGrub","type":"ConsoleProxy","cpus":1,"speed":500,"minRam":1073741824,"maxRam":1073741824,"arch":"x86_64","os":"Debian GNU/Linux 6(64-bit)","bootArgs":" template=domP type=consoleproxy host=192.168.2.201 port=8250 name=v-2-VM premium=true zone=1 pod=1 guid=Proxy.2 proxy_vm=2 disable_rp_filter=true eth2ip=192.168.2.125 eth2mask=255.255.255.0 gateway=192.168.2.1 eth0ip=169.254.1.78 eth0mask=255.255.0.0 eth1ip=192.168.2.94 eth1mask=255.255.255.0 mgmtcidr= 192.168.2.0/24 localgw=192.168.2.1 internaldns1=192.168.2.1 dns1=8.8.8.8 dns2=8.8.4.4","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"vncPassword":"38be171d61aa8901","params":{},"disks":[{"id":2,"name":"ROOT-2","mountPoint":"/export/primary","path":"58945947-d40e-4532-b26a-f97f696715f8","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"c06d70ac-1552-3d7d-afe7-07d5690d2e09","deviceId":0}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"ip":"192.168.2.125","netmask":"255.255.255.0","gateway":"192.168.2.1","mac":"06:3a:44:00:00:2e","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagged","isSecurityGroupEnabled":true},{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"ip":"169.254.1.78","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:4e","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"ip":"192.168.2.94","netmask":"255.255.255.0","gateway":"192.168.2.1","mac":"06:51:7e:00:00:0f","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false}]},"wait":0}},{"check.CheckSshCommand":{"ip":"169.254.1.78","port":3922,"interval":6,"retries":100,"name":"v-2-VM","wait":0}}] }