Hi there,

We’re in a bit of a pickle with our Cloudstack 4.4.1 install. (Yes, I know it’s 
outdated; we were hoping to upgrade soon and then this happened.) We had an 
issue where the VR for one of our Networks went down, and I mistakenly missed 
the ‘cleanup’ checkbox in the ‘Restart network’ menu of the UI. We use the VR 
solely for DNS. We now have no VRs in our setup, and the network can no longer 
be restarted due to a check that routers are running. Unfortunately, to delete 
and re-add the network, the log output states that we’d have to expunge all of 
our machines, which I am hoping to avoid doing. What are my other options? Can 
I un-mark the router as deleted in the DB, and will that allow it to limp along 
to the point where it can at least rebuild the network?

Thanks,
--Mason

(relevant logs below)

2016-09-19 11:46:50,890 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141) Executing AsyncJobVO {id:10141, 
userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: 
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd, cmdInfo: 
{"response":"json","id":"448deced-7
223-4549-98bd-5acafc811f05","sessionkey":"985SpsvETZfPpiNyTjNsOgXWVQI\u003d","cleanup":"true","ctxDetails":"{\"com.cloud.network.Network\":\"448deced-7223-4549-98bd-5acafc811f05\"}","cmdEventType":"NETWORK.RESTART","ctxUserId":"2","httpmethod":"GET","_":"1474303610834","uuid":"448deced-7223-4549-98bd-5acafc811f05","c
txAccountId":"2","ctxStartEventId":"21061"}, cmdVersion: 0, status: 
IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 
144343483243, completeMsid: null, lastUpdated: null, lastPolled: null, created: 
null}
2016-09-19 11:46:50,891 DEBUG [c.c.a.ApiServlet] (catalina-exec-22:ctx-94ea893d 
ctx-69ffa947) ===END===  192.168.42.156 -- GET  
command=restartNetwork&id=448deced-7223-4549-98bd-5acafc811f05&cleanup=true&response=json&sessionkey=985SpsvETZfPpiNyTjNsOgXWVQI%3D&_=1474303610834
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Restarting network 
204...
2016-09-19 11:46:50,925 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Shutting down the 
network id=204 as a part of network restart
2016-09-19 11:46:50,929 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 port 
forwarding rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,930 DEBUG [c.c.n.f.FirewallManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to 
forward to the network elements
2016-09-19 11:46:50,932 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 static nat 
rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,932 DEBUG [c.c.n.f.FirewallManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to 
forward to the network elements
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Public load 
balancing rules for network id=204
2016-09-19 11:46:50,934 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load 
Balancing Rules to forward to the network elements
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Revoking 0 Internal 
load balancing rules for network id=204
2016-09-19 11:46:50,936 DEBUG [c.c.n.l.LoadBalancingRulesManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no Load 
Balancing Rules to forward to the network elements
2016-09-19 11:46:50,937 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall 
ingress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,937 DEBUG [c.c.n.f.FirewallManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to 
forward to the network elements
2016-09-19 11:46:50,938 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Releasing 0 firewall 
egress rules for network id=204 as a part of shutdownNetworkRules
2016-09-19 11:46:50,939 DEBUG [c.c.n.f.FirewallManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) There are no rules to 
forward to the network elements
2016-09-19 11:46:50,941 DEBUG [c.c.n.r.RulesManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Found 0 static nat 
rules to apply for network id 204
2016-09-19 11:46:51,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network 
shutdown to SecurityGroupProvider
2016-09-19 11:46:51,032 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Sending network 
shutdown to VirtualRouter
2016-09-19 11:46:51,034 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Implementing the 
network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and 
resources as a part of network restart
2016-09-19 11:46:51,040 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking 
SecurityGroupProvider to implemenet 
Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,043 DEBUG [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Asking VirtualRouter 
to implemenet Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6]
2016-09-19 11:46:51,049 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is acquired for 
network id 204 as a part of router startup in 
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] 
: Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,051 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Lock is released for 
network id 204 as a part of router startup in 
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] 
: Dest[Zone(1)-Pod(null)-Cluster(null)-H
ost(null)-Storage()]
2016-09-19 11:46:51,053 WARN  [o.a.c.e.o.NetworkOrchestrator] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Failed to implement 
network Ntwk[448deced-7223-4549-98bd-5acafc811f05|Guest|6] elements and 
resources as a part of network restart due to
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is 
unreachable: Can't find all necessary running routers!
        at 
com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:199)
        at 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1080)
        at 
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2430)
        at 
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1892)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at 
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at 
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at 
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
        at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at 
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at 
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at 
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at 
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy156.restartNetwork(Unknown Source)
        at 
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
        at 
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
        at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
        at 
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at 
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at 
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at 
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
        at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
2016-09-19 11:46:51,054 WARN  [c.c.n.NetworkServiceImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141 ctx-6dda2ed3) Network id=204 failed 
to restart.
2016-09-19 11:46:51,072 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141) Complete async job-10141, 
jobStatus: FAILED, resultCode: 530, result: 
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
 to restart network"}
2016-09-19 11:46:51,087 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141) Done executing 
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for job-10141
2016-09-19 11:46:51,107 INFO  [o.a.c.f.j.i.AsyncJobMonitor] 
(API-Job-Executor-5:ctx-8a42cc36 job-10141) Remove job-10141 from job monitoring

Reply via email to