It looks there is no entry in template_store_ref table for the systemvm template
-Wei On Wed, 3 Aug 2022 at 21:43, [email protected] <[email protected]> wrote: > Hi everyone, > > faceing currently some challanges regarding my network configuration in CS > 4.16.1. > > Setup: > VPC with redundant routers and some tiers as well as an private gateway. > > Today i wanted to restart the whole VPC - sadly only one vrouter > "survived" .... Currently i can't depoly any networks in or outside the > VPC. Also the second router shall be delployed but keeps in the stopped > state. > I really dont have a clue where to look at first... > > here the logfile from the Management-Server: > > 2022-08-03 20:35:48,768 DEBUG [c.c.n.r.NetworkHelperImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating the VR with id=74 in datacenter Zone {"id": "1", "name": > "xxxxxxxxxx", "uuid": "48e2e928-3300-43b5-8e3a-d> > 2022-08-03 20:35:48,776 DEBUG [c.c.n.r.NetworkHelperImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Adding nic for Virtual Router in Control network > 2022-08-03 20:35:48,781 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Found existing network configuration for offering [Network Offering > [3-Control-System-Control-Network]: Ntwk[202|> > 2022-08-03 20:35:48,781 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Releasing lock for Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- > Account {"id": 1, "name": "system", "uuid> > 2022-08-03 20:35:48,785 DEBUG [c.c.n.r.NetworkHelperImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Adding nic for Virtual Router in Public network > 2022-08-03 20:35:48,789 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Found existing network configuration for offering [Network Offering > [1-Public-System-Public-Network]: Ntwk[200|Pu> > 2022-08-03 20:35:48,789 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Releasing lock for Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] -- > Account {"id": 1, "name": "system", "uuid> > 2022-08-03 20:35:48,793 INFO [c.c.n.r.NetworkHelperImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Use same MAC as previous RvR, the MAC is 1e:00:59:00:00:b2 > 2022-08-03 20:35:48,793 DEBUG [c.c.n.r.NetworkHelperImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Adding nic for Virtual Router in Guest network Ntwk[249|Guest|30] > 2022-08-03 20:35:48,801 DEBUG [c.c.u.d.T.Transaction] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Rolling back the transaction: Time = 0 Name = API-Job-Executor-60; called > by -TransactionLegacy.rollback:888-PrivateIpDa> > 2022-08-03 20:35:48,808 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=236 > 2022-08-03 20:35:48,851 INFO [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > allocating virtual machine from > template:0573ec70-0a30-4e2a-be9e-4675bf755cf2 with hostname:r-74-VM and 12 > netw> > 2022-08-03 20:35:48,852 DEBUG [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating entries for VM: VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3"> > 2022-08-03 20:35:48,857 DEBUG [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nics for VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,858 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,866 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,886 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,904 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=249 > 2022-08-03 20:35:48,907 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,917 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=236 > 2022-08-03 20:35:48,920 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,939 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=240 > 2022-08-03 20:35:48,942 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,959 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=241 > 2022-08-03 20:35:48,962 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:48,985 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=242 > 2022-08-03 20:35:48,987 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:49,004 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=243 > 2022-08-03 20:35:49,008 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:49,026 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=244 > 2022-08-03 20:35:49,029 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:49,046 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=245 > 2022-08-03 20:35:49,049 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:49,065 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=246 > 2022-08-03 20:35:49,068 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type=> > 2022-08-03 20:35:49,085 DEBUG [c.c.n.NetworkModelImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Service SecurityGroup is not supported in the network id=247 > 2022-08-03 20:35:49,088 DEBUG [c.c.v.VirtualMachineManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Allocating disks for VM instance {id: "74", name: "r-74-VM", uuid: > "09dc644b-76a2-404a-82fd-aecc8d5799c3", type> > 2022-08-03 20:35:49,088 INFO [o.a.c.e.o.VolumeOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > adding disk object ROOT-74 to r-74-VM > 2022-08-03 20:35:49,091 DEBUG [c.c.u.d.T.Transaction] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Rolling back the transaction: Time = 234 Name = API-Job-Executor-60; > called by -TransactionLegacy.rollback:888-Transacti> > 2022-08-03 20:35:49,099 WARN [o.a.c.e.o.NetworkOrchestrator] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Failed to implement network Ntwk[249|Guest|30] elements and resources as a > part of network restart due to > java.lang.NullPointerException > at > org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO.getSize(TemplateDataStoreVO.java:280) > at > com.cloud.template.TemplateManagerImpl.getTemplateSize(TemplateManagerImpl.java:2009) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > at com.sun.proxy.$Proxy201.getTemplateSize(Unknown Source) > at > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.allocateTemplatedVolume(VolumeOrchestrator.java:834) > at > org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.allocateTemplatedVolumes(VolumeOrchestrator.java:954) > at > com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:496) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25) > at > com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21) > at com.cloud.utils.db.Transaction.execute(Transaction.java:40) > at > com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:471) > at > com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528) > at > com.cloud.network.router.VpcNetworkHelperImpl.reallocateRouterNetworks(VpcNetworkHelperImpl.java:169) > at > com.cloud.network.router.NetworkHelperImpl.deployRouter(NetworkHelperImpl.java:520) > at > org.cloud.network.router.deployment.VpcRouterDeploymentDefinition.deployAllVirtualRouters(VpcRouterDeploymentDefinition.java:178) > at > org.cloud.network.router.deployment.RouterDeploymentDefinition.executeDeployment(RouterDeploymentDefinition.java:377) > at > org.cloud.network.router.deployment.RouterDeploymentDefinition.findOrDeployVirtualRouter(RouterDeploymentDefinition.java:239) > at > org.cloud.network.router.deployment.VpcRouterDeploymentDefinition.findOrDeployVirtualRouter(VpcRouterDeploymentDefinition.java:140) > at > org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205) > at > com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:215) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1488) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1423) > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3302) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2068) > at > com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2089) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) > at > java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.base/java.lang.reflect.Method.invoke(Method.java:566) > at > org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) > at > org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > at > com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) > at > org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) > at > org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) > at > org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) > at com.sun.proxy.$Proxy168.restartNetwork(Unknown Source) > at > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:98) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) > at > com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568) > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > at > java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > at java.base/java.lang.Thread.run(Thread.java:829) > 2022-08-03 20:35:49,100 WARN [c.c.n.NetworkServiceImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867) > Network id=249 failed to restart. > 2022-08-03 20:35:49,111 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Complete async > job-3342, jobStatus: FAILED, resultCode: 530, result: > org.apache.cloudstack.api.response.ExceptionResponse/nu> > 2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Publish async > job-3342 complete on message bus > 2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Wake up jobs > related to job-3342 > 2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Update db > status for job-3342 > 2022-08-03 20:35:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Wake up jobs > joined with job-3342 and disjoin all subjobs created from job- 3342 > 2022-08-03 20:35:49,119 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] > (API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Done executing > org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for > job-3342 > > Regards, > Chris >
