Michael, When you Restart the Network and tick the box that says Clean Up, the ACS will automatically remove the old VR and recreate a new one from the template.
Regarding your Stale NFS - I would investigate this and fix it prior to doing anything else. I've had cases with stale nfs where the client would have no issues accessing some images, but would not see other images. From my experience, this tend to be nfs client specific and usually doesn't effect other clients. Andrei -- Andrei Mikhailovsky Director Arhont Information Security Web: http://www.arhont.com http://www.wi-foo.com Tel: +44 (0)870 4431337 Fax: +44 (0)208 429 3111 PGP: Key ID - 0x2B3438DE PGP: Server - keyserver.pgp.com DISCLAIMER The information contained in this email is intended only for the use of the person(s) to whom it is addressed and may be confidential or contain legally privileged information. If you are not the intended recipient you are hereby notified that any perusal, use, distribution, copying or disclosure is strictly prohibited. If you have received this email in error please immediately advise us by return email at [email protected] and delete and purge the email and any attachments without making a copy. ----- Original Message ----- From: "Michael Lukzak" <[email protected]> To: [email protected] Sent: Thursday, 7 August, 2014 8:43:06 PM Subject: Re[4]: Urgent help needed. Can't start Virtual Router. I removed this host from CS, but still problem is. How I can "rebuild VR" from zero? It's looks like CS want to start VM (witch really is a VirtualRouter) that not exists on the host. INFO [xen.resource.CitrixResourceBase] (DirectAgent-148:) VM does not exist on XenServer11681b28-5f38-4f20-aaae-0c9ed1e9ecc7 Or other way. How I can bind new network to VM? Ony way is modify VM parameter in database? Michael > depending on your setup, but what I would do move all your Instances from > host machine where the VR lives and reboot. Again, that's what I would do. > you can try this link for XenServer 5 > http://blogs.citrix.com/2009/01/26/how-to-configure-an-nfs-based-iso-sr-using-local-storage-on-a-xenserver5-host/ > Thanks, > On Thu, Aug 7, 2014 at 12:17 PM, Michael Lukzak <[email protected]> wrote: >> Hi, >> We are usning xenserver from Citrix. >> I found some clue. In console to server on this host I see that nfs has >> error "Stale NFS file handle". >> How to remount NFS? Maybe this is a problem? >> But why other VR are started and few of them not... ;( >> Best regards, >> Michael >> > Hi Michael, >> > if it helps, I would restart the agent on the host machine where VR >> lives. >> > it depends on hypervisor but on KVM, following command: service >> > cloudstack-agent restart >> > Thanks, >> > Motty >> > On Thu, Aug 7, 2014 at 11:52 AM, Michael Lukzak <[email protected]> wrote: >> >> Hi, >> >> One of our host are died (after very high load). >> >> Host has a lot of Virtual Routers running. >> >> We are using advanced networking. >> >> Only way to bring machine alive was hard reset. >> >> After this CloudStack decied to start all of VirtualRouterts, >> >> when server was back online. >> >> Most of VR are started, but couple don't. >> >> Cloudstack want to start this VM, but can't ;( >> >> Here are log from this action. >> >> It's looks like that CS are looking for VR host on Xen host, but he >> >> couldn't find. >> >> Please help how to repair this situation. >> >> Restart network with clean up options ending with the same resoult. >> >> Is there any option to recreate VR? >> >> I tried to remove VR from Infrastructure -> Virtual Router, but there is >> >> still problem >> >> with create new one. >> >> Here are logs from "restart network with clean up options" >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[200|Public|1] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[338|Guest|8] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-151:job-6526) Network Ntwk[202|Control|3] is not >> associated >> >> with any VPC >> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-151:job-6526) >> >> Failed to start instance VM[DomainRouter|r-1015-VM] >> >> java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> >> at >> >> >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> >> at >> >> >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> INFO [xen.resource.CitrixResourceBase] (DirectAgent-16:) VM does not >> >> exist on XenServer0d1d5015-4061-4554-96de-406155449296 >> >> WARN [cloud.network.NetworkManagerImpl] (Job-Executor-151:job-6526) >> >> Failed to implement network Ntwk[338|Guest|8] elements and resources as >> a >> >> part of network restart due to >> >> com.cloud.exception.AgentUnavailableException: Resource [Host:9] is >> >> unreachable: Host 9: Unable to start instance due to null >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouters(VirtualNetworkApplianceManagerImpl.java:1924) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.deployVirtualRouterInGuestNetwork(VirtualNetworkApplianceManagerImpl.java:1902) >> >> at >> >> >> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:175) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:1518) >> >> at >> >> >> com.cloud.network.NetworkManagerImpl.restartNetwork(NetworkManagerImpl.java:2473) >> >> at >> >> >> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1381) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:92) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> Caused by: java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> ... 23 more >> >> WARN [cloud.network.NetworkServiceImpl] (Job-Executor-151:job-6526) >> >> Network id=338 failed to restart. >> >> WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-151:job-6526) >> Unable >> >> to unregister active job 6526 from JMX monitoring >> >> Here are log when I want to start VR (his state is stopeed, I want to >> >> start and got this) >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[200|Public|1] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[338|Guest|8] is not associated >> >> with any VPC >> >> WARN [network.element.VpcVirtualRouterElement] >> >> (Job-Executor-153:job-6528) Network Ntwk[202|Control|3] is not >> associated >> >> with any VPC >> >> ERROR [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-153:job-6528) >> >> Failed to start instance VM[DomainRouter|r-1015-VM] >> >> java.lang.NullPointerException >> >> at >> >> >> com.cloud.network.NetworkModelImpl.getIpInNetwork(NetworkModelImpl.java:763) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VirtualNetworkApplianceManagerImpl.java:2346) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeNetworkRulesForNetwork(VpcVirtualNetworkApplianceManagerImpl.java:928) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VirtualNetworkApplianceManagerImpl.java:2241) >> >> at >> >> >> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.finalizeCommandsOnStart(VpcVirtualNetworkApplianceManagerImpl.java:767) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.finalizeDeployment(VirtualNetworkApplianceManagerImpl.java:2205) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:763) >> >> at >> >> >> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:471) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.start(VirtualNetworkApplianceManagerImpl.java:2616) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startVirtualRouter(VirtualNetworkApplianceManagerImpl.java:1824) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2874) >> >> at >> >> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.startRouter(VirtualNetworkApplianceManagerImpl.java:2824) >> >> at >> >> >> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) >> >> at >> >> >> org.apache.cloudstack.api.command.admin.router.StartRouterCmd.execute(StartRouterCmd.java:103) >> >> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:162) >> >> at >> >> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) >> >> at >> >> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >> >> at >> >> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) >> >> at java.util.concurrent.FutureTask.run(FutureTask.java:166) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >> >> at >> >> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >> >> at java.lang.Thread.run(Thread.java:722) >> >> Best Regards, >> >> Michael >> -- >> Pozdrowienia, >> Michael Lukzak -- Pozdrowienia, Michael Lukzak
