Hi everyone,
faceing currently some challanges regarding my network configuration in CS
4.16.1.
Setup:
VPC with redundant routers and some tiers as well as an private gateway.
Today i wanted to restart the whole VPC - sadly only one vrouter "survived"
.... Currently i can't depoly any networks in or outside the VPC. Also the
second router shall be delployed but keeps in the stopped state.
I really dont have a clue where to look at first...
here the logfile from the Management-Server:
2022-08-03 20:35:48,768 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating the VR with id=74 in datacenter Zone {"id": "1", "name":
"xxxxxxxxxx", "uuid": "48e2e928-3300-43b5-8e3a-d>
2022-08-03 20:35:48,776 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Adding nic for Virtual Router in Control network
2022-08-03 20:35:48,781 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Found existing network configuration for offering [Network Offering
[3-Control-System-Control-Network]: Ntwk[202|>
2022-08-03 20:35:48,781 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Releasing lock for Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] --
Account {"id": 1, "name": "system", "uuid>
2022-08-03 20:35:48,785 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Adding nic for Virtual Router in Public network
2022-08-03 20:35:48,789 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Found existing network configuration for offering [Network Offering
[1-Public-System-Public-Network]: Ntwk[200|Pu>
2022-08-03 20:35:48,789 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Releasing lock for Acct[60bddbd5-1d8a-11ec-83ce-525400c9c662-system] --
Account {"id": 1, "name": "system", "uuid>
2022-08-03 20:35:48,793 INFO [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Use same MAC as previous RvR, the MAC is 1e:00:59:00:00:b2
2022-08-03 20:35:48,793 DEBUG [c.c.n.r.NetworkHelperImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Adding nic for Virtual Router in Guest network Ntwk[249|Guest|30]
2022-08-03 20:35:48,801 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Rolling back the transaction: Time = 0 Name = API-Job-Executor-60; called
by -TransactionLegacy.rollback:888-PrivateIpDa>
2022-08-03 20:35:48,808 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=236
2022-08-03 20:35:48,851 INFO [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
allocating virtual machine from
template:0573ec70-0a30-4e2a-be9e-4675bf755cf2 with hostname:r-74-VM and 12
netw>
2022-08-03 20:35:48,852 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating entries for VM: VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3">
2022-08-03 20:35:48,857 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nics for VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,858 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,866 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,886 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,904 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=249
2022-08-03 20:35:48,907 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,917 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=236
2022-08-03 20:35:48,920 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,939 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=240
2022-08-03 20:35:48,942 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,959 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=241
2022-08-03 20:35:48,962 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:48,985 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=242
2022-08-03 20:35:48,987 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:49,004 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=243
2022-08-03 20:35:49,008 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:49,026 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=244
2022-08-03 20:35:49,029 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:49,046 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=245
2022-08-03 20:35:49,049 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:49,065 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=246
2022-08-03 20:35:49,068 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating nic for vm VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type=>
2022-08-03 20:35:49,085 DEBUG [c.c.n.NetworkModelImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Service SecurityGroup is not supported in the network id=247
2022-08-03 20:35:49,088 DEBUG [c.c.v.VirtualMachineManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Allocating disks for VM instance {id: "74", name: "r-74-VM", uuid:
"09dc644b-76a2-404a-82fd-aecc8d5799c3", type>
2022-08-03 20:35:49,088 INFO [o.a.c.e.o.VolumeOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
adding disk object ROOT-74 to r-74-VM
2022-08-03 20:35:49,091 DEBUG [c.c.u.d.T.Transaction]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Rolling back the transaction: Time = 234 Name = API-Job-Executor-60;
called by -TransactionLegacy.rollback:888-Transacti>
2022-08-03 20:35:49,099 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Failed to implement network Ntwk[249|Guest|30] elements and resources as a
part of network restart due to
java.lang.NullPointerException
at
org.apache.cloudstack.storage.datastore.db.TemplateDataStoreVO.getSize(TemplateDataStoreVO.java:280)
at
com.cloud.template.TemplateManagerImpl.getTemplateSize(TemplateManagerImpl.java:2009)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy201.getTemplateSize(Unknown Source)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.allocateTemplatedVolume(VolumeOrchestrator.java:834)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.allocateTemplatedVolumes(VolumeOrchestrator.java:954)
at
com.cloud.vm.VirtualMachineManagerImpl$1.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:496)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:25)
at
com.cloud.utils.db.TransactionCallbackWithExceptionNoReturn.doInTransaction(TransactionCallbackWithExceptionNoReturn.java:21)
at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
at
com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:471)
at
com.cloud.vm.VirtualMachineManagerImpl.allocate(VirtualMachineManagerImpl.java:528)
at
com.cloud.network.router.VpcNetworkHelperImpl.reallocateRouterNetworks(VpcNetworkHelperImpl.java:169)
at
com.cloud.network.router.NetworkHelperImpl.deployRouter(NetworkHelperImpl.java:520)
at
org.cloud.network.router.deployment.VpcRouterDeploymentDefinition.deployAllVirtualRouters(VpcRouterDeploymentDefinition.java:178)
at
org.cloud.network.router.deployment.RouterDeploymentDefinition.executeDeployment(RouterDeploymentDefinition.java:377)
at
org.cloud.network.router.deployment.RouterDeploymentDefinition.findOrDeployVirtualRouter(RouterDeploymentDefinition.java:239)
at
org.cloud.network.router.deployment.VpcRouterDeploymentDefinition.findOrDeployVirtualRouter(VpcRouterDeploymentDefinition.java:140)
at
org.cloud.network.router.deployment.RouterDeploymentDefinition.deployVirtualRouter(RouterDeploymentDefinition.java:205)
at
com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:215)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElements(NetworkOrchestrator.java:1488)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1423)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:3302)
at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2068)
at
com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:2089)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy168.restartNetwork(Unknown Source)
at
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:98)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-03 20:35:49,100 WARN [c.c.n.NetworkServiceImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342 ctx-150e61a2) (logid:6b6b9867)
Network id=249 failed to restart.
2022-08-03 20:35:49,111 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Complete async
job-3342, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/nu>
2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Publish async
job-3342 complete on message bus
2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Wake up jobs
related to job-3342
2022-08-03 20:35:49,112 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Update db
status for job-3342
2022-08-03 20:35:49,114 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Wake up jobs
joined with job-3342 and disjoin all subjobs created from job- 3342
2022-08-03 20:35:49,119 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-60:ctx-a9f63190 job-3342) (logid:6b6b9867) Done executing
org.apache.cloudstack.api.command.user.network.RestartNetworkCmd for
job-3342
Regards,
Chris
