What was the solution to this??
Asmita Vagyani <Asmita.Vagyani@...> writes: > > Hi all, > > I have re-installed the CS4 on RHEL server. > The cluster and its hosts are added properly, and in running state. > SSVMs area also in running state. > > When I create an instance from console I get the following error in management console logs: > Any idea abt this issue? (I saw link http://bugs.cloudstack.org/browse/CS- 14465 , but I don't see the > /systemvm folder created on the /mnt/secondary folder) > > 2013-01-23 01:06:44,894 ERROR [vmware.resource.VmwareResource] > (DirectAgent-108:torvms03lx.sigmasys.net) Unable to execute NetworkUsage command on DomR > (10.0.100.182), domR may not be ready yet. failure due to Exception: java.lang.Exception > Message: Failed to authentication SSH user root on host 10.0.100.182 > > java.lang.Exception: Failed to authentication SSH user root on host 10.0.100.182 > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:144) > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:37) > at com.cloud.hypervisor.vmware.resource.VmwareResource.networkUsage(VmwareResou rce.java:4566) > at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j ava:1911) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes ource.java:425) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java: 191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201(ScheduledThreadPoolExecutor.java:178) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:292) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 10) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 03) > at java.lang.Thread.run(Thread.java:722) > 2013-01-23 01:06:44,899 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-108:torvms03lx.sigmasys.net) Executing resource GetDomRVersionCmd: {"accessDetails":{"router.name":"r-5- VM","router.ip":"10.0.100.182"},"wait":0} > 2013-01-23 01:06:44,899 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-108:torvms03lx.sigmasys.net) Run command on domR 10.0.100.182, /opt/cloud/bin/get_template_version.sh > 2013-01-23 01:06:44,899 DEBUG [vmware.resource.VmwareResource] > (DirectAgent-108:torvms03lx.sigmasys.net) Use router's private IP for SSH control. IP : 10.0.100.182 > 2013-01-23 01:06:45,138 ERROR [utils.ssh.SshHelper] (DirectAgent- 108:torvms03lx.sigmasys.net) > Failed to authentication SSH user root on host 10.0.100.182 > 2013-01-23 01:06:45,139 ERROR [vmware.resource.VmwareResource] > (DirectAgent-108:torvms03lx.sigmasys.net) GetDomRVersionCmd failed due to Exception: java.lang.Exception > Message: Failed to authentication SSH user root on host 10.0.100.182 > > java.lang.Exception: Failed to authentication SSH user root on host 10.0.100.182 > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:144) > at com.cloud.utils.ssh.SshHelper.sshExecute(SshHelper.java:37) > at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.j ava:1753) > at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareRes ource.java:433) > at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java: 191) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$ 201(ScheduledThreadPoolExecutor.java:178) > at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Sch eduledThreadPoolExecutor.java:292) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 10) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 03) > at java.lang.Thread.run(Thread.java:722) > 2013-01-23 01:06:45,143 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-108:null) Seq > 2-2082210641: Cancelling because one of the answers is false and it is stop on error. > 2013-01-23 01:06:45,143 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-108:null) Seq > 2-2082210641: Response Received: > 2013-01-23 01:06:45,144 DEBUG [agent.transport.Request] (DirectAgent- 108:null) Seq 2-2082210641: > Processing: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, > [{"StartAnswer":{"vm":{"id":5,"name":"r-5- VM","bootloader":"HVM","type":"DomainRouter","cpus":1,"speed":500,"minRam":1 34217728,"maxRam":134217728,"arch":"i686","os":"Debian > GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-5-VM eth0ip=10.0.100.182 > eth0mask=255.255.255.0 gateway=10.0.100.1 domain=cs1cloud.internal dhcprange=10.0.100.1 > eth1ip=0.0.0.0 eth1mask=0.0.0.0 sshonguest=true type=dhcpsrvr disable_rp_filter=true > extra_pubnics=2 dns1=10.0.100.5 > nic_macs=06:12:00:00:00:08|02:00:4f:6b:00:08","rebootOnCrash":false,"enableH A":true,"limitCpuUse":false,"vncPassword":"b8fdaa863d476636","params": {"nicAdapter":"E1000"},"uuid":"04ee40b9-bd06-4814-b01e- aa4928da5d96","disks":[{"id":6,"name":"ROOT- 5","mountPoint":"/CloudStack/tormsa02-raid0-4","path":"ROOT- 5","size":2097152000,"type":"ROOT","storagePoolType":"VMFS","storagePoolUuid ":"3546bfef-40ed-3f8f-9484-b5cd712235f8","deviceId":0}],"nics": [{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"e04d295a- e633-407d-96e0- aef8550df2d4","ip":"10.0.100.182","netmask":"255.255.255.0","gateway":"10.0. 100.1","mac":"06:12:00:00:00:08","dns1":"10.0.100.5","broadcastType":"Vlan", "type":"Guest","broadcastUri":"vlan://untagged","isolationUri":"ec2://untagg ed","isSecurityGroupEnabled":false}, {"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"31e1d523-79da- 40f7-b909- 47a0cce3e8cb","ip":"0.0.0.0","netmask":"0.0.0.0","gateway":"0.0.0.0","mac":" 02:00:4f:6b:00:08","broadcastType":"LinkLocal","type":"Control","isSecurityG roupEnabled":false}]},"result":true,"wait":0}},{"check.CheckSshAnswer": {"result":true,"wait":0}},{"GetDomRVersionAnswer": {"result":false,"details":"GetDomRVersionCmd > failed due to Exception: java.lang.Exception\nMessage: Failed to authentication SSH user root on host > 10.0.100.182\n","wait":0}}] } > 2013-01-23 01:06:45,144 DEBUG [agent.manager.AgentAttache] (DirectAgent- 108:null) Seq > 2-2082210641: No more commands found > 2013-01-23 01:06:45,144 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210641: Received: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, { StartAnswer, > CheckSshAnswer, GetDomRVersionAnswer } } > 2013-01-23 01:06:45,282 WARN [network.router.VirtualNetworkApplianceManagerImpl] > (Job-Executor-7:job-13) Unable to get the template/scripts version of router r-5-VM due to: > GetDomRVersionCmd failed due to Exception: java.lang.Exception > Message: Failed to authentication SSH user root on host 10.0.100.182 > > 2013-01-23 01:06:45,283 INFO [cloud.vm.VirtualMachineManagerImpl] (Job- Executor-7:job-13) The > guru did not like the answers so stopping VM[DomainRouter|r-5-VM] > 2013-01-23 01:06:45,287 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210644: Sending { Cmd , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"r-5-VM","wait":0}}] } > 2013-01-23 01:06:45,287 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210644: Executing: { Cmd , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"r-5-VM","wait":0}}] } > 2013-01-23 01:06:45,287 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-182:null) Seq > 2-2082210644: Executing request > 2013-01-23 01:06:45,288 INFO [vmware.resource.VmwareResource] > (DirectAgent-182:torvms03lx.sigmasys.net) Executing resource StopCommand: {"isProxy":false,"vmName":"r-5-VM","wait":0} > 2013-01-23 01:06:45,288 DEBUG [vmware.mo.HostMO] (DirectAgent- 182:torvms03lx.sigmasys.net) find > VM r-5-VM on host > 2013-01-23 01:06:45,288 DEBUG [vmware.mo.HostMO] (DirectAgent- 182:torvms03lx.sigmasys.net) load > VM cache on host > 2013-01-23 01:06:46,211 INFO [vmware.resource.VmwareResource] > (DirectAgent-182:torvms03lx.sigmasys.net) Remove all snapshot before stopping VM r-5-VM > 2013-01-23 01:06:46,595 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) > VmStatsCollector is running... > 2013-01-23 01:06:49,438 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] > (RouterStatusMonitor-1:null) Found 0 routers. > 2013-01-23 01:06:53,877 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-8:null) > Ping from 6 > 2013-01-23 01:06:54,281 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-182:null) Seq > 2-2082210644: Response Received: > 2013-01-23 01:06:54,282 DEBUG [agent.transport.Request] (DirectAgent- 182:null) Seq 2-2082210644: > Processing: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, > [{"StopAnswer":{"vncPort":0,"result":true,"details":"Stop VM r-5-VM Succeed","wait":0}}] } > 2013-01-23 01:06:54,282 DEBUG [agent.manager.AgentAttache] (DirectAgent- 182:null) Seq > 2-2082210644: No more commands found > 2013-01-23 01:06:54,282 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210644: Received: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, { StopAnswer } } > 2013-01-23 01:06:54,282 DEBUG [agent.manager.AgentManagerImpl] (Job- Executor-7:job-13) Details > from executing class com.cloud.agent.api.StopCommand: Stop VM r-5-VM Succeed > 2013-01-23 01:06:54,282 ERROR [cloud.vm.VirtualMachineManagerImpl] (Job- Executor-7:job-13) > Failed to start instance VM[DomainRouter|r-5-VM] > com.cloud.utils.exception.ExecutionException: Unable to start VM[DomainRouter|r-5-VM] due to > error in finalizeStart, not retrying > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp l.java:814) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java: 472) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet workApplianceManagerImpl.java:2352) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout er(VirtualNetworkApplianceManagerImpl.java:1597) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir tualNetworkApplianceManagerImpl.java:1697) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1675) > at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement. java:205) > at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java: 2056) > at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2172 ) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2113) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp l.java:752) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java: 472) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:29 29) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:26 01) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:25 89) > at com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36) > at com.cloud.api.commands.DeployVMCmd.execute(DeployVMCmd.java:330) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:432) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11 10) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6 03) > at java.lang.Thread.run(Thread.java:722) > 2013-01-23 01:06:54,797 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job- Executor-7:job-13) > Cleaning up resources for the vm VM[DomainRouter|r-5-VM] in Starting state > 2013-01-23 01:06:54,798 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210645: Sending { Cmd , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"r-5-VM","wait":0}}] } > 2013-01-23 01:06:54,969 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210645: Executing: { Cmd , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"r-5-VM","wait":0}}] } > 2013-01-23 01:06:54,969 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-173:null) Seq > 2-2082210645: Executing request > 2013-01-23 01:06:54,969 INFO [vmware.resource.VmwareResource] > (DirectAgent-173:torvms03lx.sigmasys.net) Executing resource StopCommand: {"isProxy":false,"vmName":"r-5-VM","wait":0} > 2013-01-23 01:06:54,969 DEBUG [vmware.mo.HostMO] (DirectAgent- 173:torvms03lx.sigmasys.net) find > VM r-5-VM on host > 2013-01-23 01:06:54,969 DEBUG [vmware.mo.HostMO] (DirectAgent- 173:torvms03lx.sigmasys.net) load > VM cache on host > 2013-01-23 01:06:55,878 INFO [vmware.resource.VmwareResource] > (DirectAgent-173:torvms03lx.sigmasys.net) VM r-5-VM is already in stopped state > 2013-01-23 01:06:55,879 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-173:null) Seq > 2-2082210645: Response Received: > 2013-01-23 01:06:55,879 DEBUG [agent.transport.Request] (DirectAgent- 173:null) Seq 2-2082210645: > Processing: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, > [{"StopAnswer":{"vncPort":0,"result":true,"details":"VM r-5-VM is already in stopped > state","wait":0}}] } > 2013-01-23 01:06:55,879 DEBUG [agent.manager.AgentAttache] (DirectAgent- 173:null) Seq > 2-2082210645: No more commands found > 2013-01-23 01:06:55,879 DEBUG [agent.transport.Request] (Job-Executor- 7:job-13) Seq > 2-2082210645: Received: { Ans: , MgmtId: 345051758635, via: 2, Ver: v1, Flags: 110, { StopAnswer } } > > 2013-01-23 01:06:56,964 DEBUG [cloud.capacity.CapacityManagerImpl] (Job- Executor-7:job-13) > release cpu from host: 2, old used: 1500,reserved: 0, actual total: 48000, total with overprovisioning: > 48000; new used: 1000,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2013-01-23 01:06:56,964 DEBUG [cloud.capacity.CapacityManagerImpl] (Job- Executor-7:job-13) > release mem from host: 2, old used: 1476395008,reserved: 0, total: 34357202944; new used: > 1342177280,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2013-01-23 01:06:56,986 INFO [cloud.vm.VirtualMachineManagerImpl] (Job- Executor-7:job-13) > Unable to contact resource. > com.cloud.exception.AgentUnavailableException: Resource [Host:2] is unreachable: Host 2: Unable to > start instance due to Unable to start VM[DomainRouter|r-5-VM] due to error in finalizeStart, not retrying > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp l.java:847) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java: 472) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNet workApplianceManagerImpl.java:2352) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRout er(VirtualNetworkApplianceManagerImpl.java:1597) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(Vir tualNetworkApplianceManagerImpl.java:1697) > at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRou terInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1675) > at com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement. java:205) > at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java: 2056) > at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2172 ) > at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2113) > at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImp l.java:752) > at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java: 472) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:29 29) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:26 01) > at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:25 89) > at com.cloud.event.ActionEventCallback.intercept(ActionEventCallback.java:36) > > Thanks and Regards. > > Asmita Patil Vagyani. > >
