I removed this host from CS, but still problem is. How I can "rebuild VR" from zero? It's looks like CS want to start VM (witch really is a VirtualRouter) that not exists on the host.
INFO [xen.resource.CitrixResourceBase] (DirectAgent-148:) VM does not exist on XenServer11681b28-5f38-4f20-aaae-0c9ed1e9ecc7 Or other way. How I can bind new network to VM? Ony way is modify VM parameter in database? Michael > depending on your setup, but what I would do move all your Instances from > host machine where the VR lives and reboot. Again, that's what I would do. > you can try this link for XenServer 5 > http://blogs.citrix.com/2009/01/26/how-to-configure-an-nfs-based-iso-sr-using-local-storage-on-a-xenserver5-host/ > Thanks, > On Thu, Aug 7, 2014 at 12:17 PM, Michael Lukzak <[email protected]> wrote: >> Hi, >> We are usning xenserver from Citrix. >> I found some clue. In console to server on this host I see that nfs has >> error "Stale NFS file handle". >> How to remount NFS? Maybe this is a problem? >> But why other VR are started and few of them not... ;( >> Best regards, >> Michael >> > Hi Michael, >> > if it helps, I would restart the agent on the host machine where VR >> lives. >> > it depends on hypervisor but on KVM, following command: service >> > cloudstack-agent restart >> > Thanks, >> > Motty >> > On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <[email protected]> wrote: >> >> Hi, >> >> One of our host are died (after very high load). >> >> Host has a lot of Virtual Routers running. >> >> We are using advanced networking. >> >> Only way to bring machine alive was hard reset. >> >> After this CloudStack decied to start all of VirtualRouterts, >> >> when server was back online. >> >> Most of VR are started, but couple don't. >> >> Cloudstack want to start this VM, but can't ;( >> >> Here are log from this action. >> >> It's looks like that CS are looking for VR host on Xen host, but he >> >> couldn't find. >> >> Please help how to repair this situation. >> >> Restart network with clean up options ending with the same resoult. >> >> Is there any option to recreate VR? >> >> I tried to remove VR from Infrastructure -> Virtual Router, but there is >> >> still problem >> >> with create new one. >> >> Here are logs from "restart network with clean up options" >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not >> associated >> >> with any VPC >> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) >> >> Failed to start instance VM[DomainRouter|r-1015-VM] >> >> java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> >> at >> >> >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> >> at >> >> >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not >> >> exist on XenServer0d1d5015-4061-4554-96de-406155449296 >> >> WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) >> >> Failed to implement network Ntwk[338|Guest|8] elements and resources as >> a >> >> part of network restart due to >> >> com.cloud.exception.AgentUnavailableException: Resource [Host:9] is >> >> unreachable: Host 9: Unable to start instance due to null >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> >> at >> >> >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> >> at >> >> >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> Caused by: java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> ... 23 more >> >> WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) >> >> Network id=338 failed to restart. >> >> WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) >> Unable >> >> to unregister active job 6526 from JMX monitoring >> >> Here are log when I want to start VR (his state is stopeed, I want to >> >> start and got this) >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not >> associated >> >> with any VPC >> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) >> >> Failed to start instance VM[DomainRouter|r-1015-VM] >> >> java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> Best Regards, >> >> Michael >> -- >> Pozdrowienia, >> Michael Lukzak -- Pozdrowienia, Michael Lukzak
