Hi Michael, if it helps, I would restart the agent on the host machine where VR lives.
it depends on hypervisor but on KVM, following command: service cloudstack-agent restart Thanks, Motty On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <[email protected]> wrote: > Hi, > > One of our host are died (after very high load). > Host has a lot of Virtual Routers running. > > We are using advanced networking. > > Only way to bring machine alive was hard reset. > > After this CloudStack decied to start all of VirtualRouterts, > when server was back online. > > Most of VR are started, but couple don't. > Cloudstack want to start this VM, but can't ;( > Here are log from this action. > > It's looks like that CS are looking for VR host on Xen host, but he > couldn't find. > > Please help how to repair this situation. > Restart network with clean up options ending with the same resoult. > > Is there any option to recreate VR? > I tried to remove VR from Infrastructure -> Virtual Router, but there is > still problem > with create new one. > > > > Here are logs from "restart network with clean up options" > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated > with any VPC > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated > with any VPC > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not associated > with any VPC > ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) > Failed to start instance VM[DomainRouter|r-1015-VM] > java.lang.NullPointerException > at > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) > at > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) > at > com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) > at > com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not > exist on XenServer0d1d5015-4061-4554-96de-406155449296 > WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) > Failed to implement network Ntwk[338|Guest|8] elements and resources as a > part of network restart due to > com.cloud.exception.AgentUnavailableException: Resource [Host:9] is > unreachable: Host 9: Unable to start instance due to null > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) > at > com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) > at > com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) > at > com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.NullPointerException > at > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > ... 23 more > WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) > Network id=338 failed to restart. > WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) Unable > to unregister active job 6526 from JMX monitoring > > > Here are log when I want to start VR (his state is stopeed, I want to > start and got this) > > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated > with any VPC > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated > with any VPC > WARN [network.element.VpcVirtualRouterElement] > (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not associated > with any VPC > ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) > Failed to start instance VM[DomainRouter|r-1015-VM] > java.lang.NullPointerException > at > com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) > at > com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) > at > com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) > at > com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874) > at > com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > > > Best Regards, > Michael
