well... if you can restore DB that would be great - if not....
On Fri, 29 Mar 2019 at 15:19, Fariborz Navidan <mdvlinqu...@gmail.com>
wrote:
> Not for me! I think my CS DB is inconsistent somehow! I cannot figure out
> which tables should I manipulate to fix it.
>
> On Fri, Mar 29, 2019 at 6:45 PM Ivan Kudryavtsev <kudryavtsev...@bw-sw.com
> >
> wrote:
>
> > VR deletion is OK for Basic Zone in 4.11.2, work normally, VR is created
> > automatically.
> >
> > пт, 29 мар. 2019 г., 10:04 Fariborz Navidan <mdvlinqu...@gmail.com>:
> >
> > > There should be inconsistency in DB. Because I did a wrong before and
> > have
> > > deleted all records in domain_router and router_network_ref tables
> > > manually. I thought it will cause CS to think network is not
> implemented
> > > yet and will re-implement but it didn't. Indeed it still shows in logs
> > > "Asking VirtualRouter to implement network...". "Setup" state of the
> > > network is preventing from re-creation of router. What can I chnage
> state
> > > of network to so that cause CS to create new router for it?
> > >
> > > Thanks
> > >
> > >
> > > On Fri, Mar 29, 2019 at 5:12 PM Fariborz Navidan <
> mdvlinqu...@gmail.com>
> > > wrote:
> > >
> > > > 4.11.2 with KVM.
> > > >
> > > > On Fri, Mar 29, 2019 at 5:08 PM Andrija Panic <
> andrija.pa...@gmail.com
> > >
> > > > wrote:
> > > >
> > > >> cloudstack version, hypervisor (version) ?
> > > >>
> > > >> On Fri, 29 Mar 2019 at 13:34, Fariborz Navidan <
> mdvlinqu...@gmail.com
> > >
> > > >> wrote:
> > > >>
> > > >> > I did marked router vm instance to Stopped and set remoed to NULL.
> > In
> > > >> > infrastructure overview it shows 1 virtual router but clicking on
> > > View,
> > > >> > shows no data. Then I restarted the network with clleanup, It said
> > > >> > [Datacenter:1] is unreachable and automatically marked the vm
> > instance
> > > >> to
> > > >> > destroyed and removed it (set removed field). But it did not
> create
> > > the
> > > >> > router again!
> > > >> >
> > > >> > On Fri, Mar 29, 2019 at 4:46 PM Thomas Joseph <
> > thomas.jo...@gmail.com
> > > >
> > > >> > wrote:
> > > >> >
> > > >> > > They running the restart network with cleanup=true
> > > >> > >
> > > >> > > On Fri, 29 Mar 2019, 10:39 am Fariborz Navidan, <
> > > >> mdvlinqu...@gmail.com>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > It fails to restart networl. Log says it cannot find virtual
> > > router.
> > > >> > > Bellow
> > > >> > > > is the log:
> > > >> > > >
> > > >> > > > 2019-03-29 10:53:41,621 DEBUG [c.c.c.ClusterManagerImpl]
> > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Detected
> > > >> management
> > > >> > > > node left, id:8, nodeIP:178.33.230.41
> > > >> > > > 2019-03-29 10:53:41,621 INFO [c.c.c.ClusterManagerImpl]
> > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Trying to
> > > >> connect
> > > >> > to
> > > >> > > > 178.33.230.41
> > > >> > > > 2019-03-29 10:53:41,621 INFO [c.c.c.ClusterManagerImpl]
> > > >> > > > (Cluster-Heartbeat-1:ctx-277a1d9d) (logid:9902f1d7) Management
> > > node
> > > >> 8
> > > >> > is
> > > >> > > > detected inactive by timestamp but is pingable
> > > >> > > > 2019-03-29 10:53:42,526 DEBUG [c.c.a.ApiServlet]
> > > >> > > > (qtp788117692-21:ctx-219919fa) (logid:c3be28b0) ===START===
> > > >> > 2.190.177.97
> > > >> > > > -- GET
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
> > > >> > > > 2019-03-29 10:53:42,528 DEBUG [c.c.a.ApiServer]
> > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0)
> > CIDRs
> > > >> from
> > > >> > > > which account
> 'Acct[27cd01ef-3907-11e9-87ab-a4bf012ed1a6-admin]'
> > > is
> > > >> > > allowed
> > > >> > > > to perform API calls: 0.0.0.0/0,::/0
> > > >> > > > 2019-03-29 10:53:42,536 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:b95430aa)
> Add
> > > >> > > job-16321
> > > >> > > > into job monitoring
> > > >> > > > 2019-03-29 10:53:42,539 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0)
> > > submit
> > > >> > async
> > > >> > > > job-16321, details: AsyncJobVO {id:16321, userId: 2,
> accountId:
> > 2,
> > > >> > > > instanceType: None, instanceId: null, cmd:
> > > >> > > >
> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
> > > >> > > cmdInfo:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
> > > >> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> > resultCode:
> > > 0,
> > > >> > > > result: null, initMsid: 279278805450993, completeMsid: null,
> > > >> > lastUpdated:
> > > >> > > > null, lastPolled: null, created: null}
> > > >> > > > 2019-03-29 10:53:42,540 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321) (logid:155f349b)
> > > >> Executing
> > > >> > > > AsyncJobVO {id:16321, userId: 2, accountId: 2, instanceType:
> > None,
> > > >> > > > instanceId: null, cmd:
> > > >> > > >
> > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd,
> > > >> > > cmdInfo:
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> {"makeredundant":"false","cleanup":"false","response":"json","ctxUserId":"2","httpmethod":"GET","ctxStartEventId":"3200","id":"6e644551-5aee-4c9b-a75f-134f544ee97c","ctxDetails":"{\"interface
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.network.Network\":\"6e644551-5aee-4c9b-a75f-134f544ee97c\"}","ctxAccountId":"2","uuid":"6e644551-5aee-4c9b-a75f-134f544ee97c","cmdEventType":"NETWORK.RESTART","_":"1553853224533"},
> > > >> > > > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
> > resultCode:
> > > 0,
> > > >> > > > result: null, initMsid: 279278805450993, completeMsid: null,
> > > >> > lastUpdated:
> > > >> > > > null, lastPolled: null, created: null}
> > > >> > > > 2019-03-29 10:53:42,541 DEBUG [c.c.a.ApiServlet]
> > > >> > > > (qtp788117692-21:ctx-219919fa ctx-3b77ed02) (logid:c3be28b0)
> > > >> ===END===
> > > >> > > > 2.190.177.97 -- GET
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> command=restartNetwork&response=json&id=6e644551-5aee-4c9b-a75f-134f544ee97c&cleanup=false&makeredundant=false&_=1553853224533
> > > >> > > > 2019-03-29 10:53:42,551 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
> > > >> > (logid:155f349b)
> > > >> > > > Restarting network 204...
> > > >> > > > 2019-03-29 10:53:42,552 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
> > > >> > (logid:155f349b)
> > > >> > > > Implementing the network Ntwk[204|Guest|7] elements and
> > resources
> > > >> as a
> > > >> > > part
> > > >> > > > of network restart without cleanup
> > > >> > > > 2019-03-29 10:53:42,554 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
> > > >> > (logid:155f349b)
> > > >> > > > Asking VirtualRouter to implemenet Ntwk[204|Guest|7]
> > > >> > > > 2019-03-29 10:53:42,558 ERROR
> > > [o.c.n.r.d.RouterDeploymentDefinition]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
> > > >> > (logid:155f349b)
> > > >> > > > Didn't support redundant virtual router without public
> network!
> > > >> > > > 2019-03-29 10:53:42,560 WARN [o.a.c.e.o.NetworkOrchestrator]
> > > >> > > > (API-Job-Executor-2:ctx-5a2a6bbe job-16321 ctx-a30b75bf)
> > > >> > (logid:155f349b)
> > > >> > > > Failed to implement network Ntwk[204|Guest|7] elements and
> > > resources
> > > >> > as a
> > > >> > > > part of network restart due to
> > > >> > > > com.cloud.exception.ResourceUnavailableException: Resource
> > > >> > [DataCenter:1]
> > > >> > > > is unreachable: Can't find all necessary running routers!
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.network.element.VirtualRouterElement.implement(VirtualRouterElement.java:243)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1203)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1138)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2862)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1883)
> > > >> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > >> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:174)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
> > > >> > > > at com.sun.proxy.$Proxy229.restartNetwork(Unknown Source)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:99)
> > > >> > > > at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> > > >> > > > at
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
> > > >> > > > at
> > > >> > >
> > > >>
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > >> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > >> > > > at
> > > >> > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > >> > > > at java.lang.Thread.run(Thread.java:748)
> > > >> > > >
> > > >> > > > On Fri, Mar 29, 2019 at 11:08 AM Thomas Joseph <
> > > >> thomas.jo...@gmail.com
> > > >> > >
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Restart the network.
> > > >> > > > >
> > > >> > > > > On Thu, 28 Mar 2019, 10:59 pm Fariborz Navidan, <
> > > >> > mdvlinqu...@gmail.com
> > > >> > > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hello,
> > > >> > > > > > I stopped and deleted virtual router to see how it affects
> > the
> > > >> > > changed
> > > >> > > > > > network settings. I tought CS will recreate it
> automatically
> > > >> but it
> > > >> > > > > didn't
> > > >> > > > > > and now I cannot start new VMs because it complains about
> > > >> > > insufficient
> > > >> > > > > data
> > > >> > > > > > capacity. How do I force CS to re-create the router?
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >>
> > > >> --
> > > >>
> > > >> Andrija Panić
> > > >>
> > > >
> > >
> >
>
--
Andrija Panić
