I would do the following: - stop the mgmt server(s), but before that, check if the network of that offending VM (i-2-67-VM) has ANY VR, since logs say "Can't find at least one running router" - edit the DB, vm_instance table set state to "Stopped" where instance name is i-2-67-VM - ensure that specific VM is really stopped (not running on any hypervisor node) or force stop in on hypervisor side - if in the first step, you have observed that a VR exist for the given network, then in the same vm_instance table make sure that the VR is in some final state (Stopped, Running) i.e. not in a transient state like Staring or Stopping etc. If the VR doesn't exist, leave it as it is. - start the mgmt server - if previously the VR existed - restart the offending network with Clean Up - this would destroy the old VR and create a new one and then start the VM -- if previously the VR did not exist, then simply start the offending VM i-2-67-VM and it will trigger creation of the VR.
Best, On Tue, 23 Jun 2020 at 17:02, Jagdish Patil <[email protected]> wrote: > Hi Everyone, > > Recently my Cloudstack setup machine's got rebooted. After the reboot, the > virtual router is not getting created, due to which I am not able to create > or access any machine. Pasting error log below: > *CloudStack Version: **4.11.3.0* > > 2020-06-23 20:23:16,599 ERROR [o.c.n.r.d.RouterDeploymentDefinition] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) > Didn't support redundant virtual router without public network! > 2020-06-23 20:23:16,599 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Unab > le to contact resource. > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one running router! > at > > com.cloud.network.element.VirtualRouterElement.prepare(VirtualRouterElement.java:280) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareElement(NetworkOrchestrator.java:1296) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1631) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1565) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1111) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100) > > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > 2020-06-23 20:23:16,621 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Clea > ning up resources for the vm VM[User|i-2-67-VM] in Starting state > 2020-06-23 20:23:16,628 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Seq 5-62628182318120 > 96597: Sending { Cmd , MgmtId: 200242944302310, via: 5(master), Ver: v1, > Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"checkBe > > foreCleanup":false,"forceStop":false,"volumesToDisconnect":[],"vmName":"i-2-67-VM","executeInSequence":false,"wait":0}}] > } > 2020-06-23 20:23:16,836 DEBUG [c.c.a.t.Request] > (AgentManager-Handler-5:null) (logid:) Seq 5-6262818231812096597: > Processing: { Ans: , MgmtId: 200242 > 944302310, via: 5, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } > 2020-06-23 20:23:16,836 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Seq 5-62628182318120 > 96597: Received: { Ans: , MgmtId: 200242944302310, via: 5(master), Ver: > v1, Flags: 10, { StopAnswer } } > 2020-06-23 20:23:16,862 ERROR [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Failed > to remove dhcp-dns entry due to: > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > at > > com.cloud.network.element.VirtualRouterElement.removeDhcpEntry(VirtualRouterElement.java:961) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.cleanupNicDhcpDnsEntry(NetworkOrchestrator.java:2933) > > at > com.cloud.vm.UserVmManagerImpl.finalizeStop(UserVmManagerImpl.java:4389) > at > > com.cloud.vm.VirtualMachineManagerImpl.sendStop(VirtualMachineManagerImpl.java:1485) > > at > > com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1519) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100) > > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > 2020-06-23 20:23:16,876 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Changi > ng active number of nics for network id=204 on -1 > 2020-06-23 20:23:16,902 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Asking > SecurityGroupProvider to release > > NicProfile[117-67-90f86a64-4697-47df-a5fb-bef1328f3449-192.168.0.94-vlan://untagged > > 2020-06-23 20:23:16,902 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Asking > VirtualRouter to release > > NicProfile[117-67-90f86a64-4697-47df-a5fb-bef1328f3449-192.168.0.94-vlan://untagged > > 2020-06-23 20:23:16,908 WARN [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-17:ctx-f565e932 job-621/job-623 ctx-770b7937) > (logid:6812a0da) Unab > le to release some network resources. > com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] > is unreachable: Can't find at least one router! > at > > com.cloud.network.element.VirtualRouterElement.removeDhcpEntry(VirtualRouterElement.java:961) > > at > > com.cloud.network.element.VirtualRouterElement.release(VirtualRouterElement.java:898) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.releaseNic(NetworkOrchestrator.java:1919) > > at > > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.release(NetworkOrchestrator.java:1845) > > at > > com.cloud.vm.VirtualMachineManagerImpl.cleanup(VirtualMachineManagerImpl.java:1567) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1272) > > at > > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4937) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > > at > > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5100) > > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > > at > > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > > at > > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > > at > > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529) > > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > Can someone please help me with creating the virtual router? > Thanks and Regards, > Jagdish Patil > -- Andrija Panić
