[
https://issues.apache.org/jira/browse/CLOUDSTACK-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113378#comment-14113378
]
Jayapal Reddy commented on CLOUDSTACK-7424:
-------------------------------------------
>From the logs, the router is 'stopping' state and network restart operation is
>performed.
'outer r-229-VM is in Stopping, so not sending apply static nat rules commands
to the backend'
2014-08-24 07:31:03,671 WARN [o.a.c.e.o.NetworkOrchestrator]
(API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Unable to complete
shutdown of the network elements due to element: VpcVirtualRouter
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
unreachable: Unable to remove vm VM[DomainRouter|r-229-VM] from network, is not
in the right state
So please update the automation script to check router state before performing
network state.
Also please check script operation for what caused the router 'stopping' sate.
Thanks,
Jayapal
> [Automation] Failed to restart network
> --------------------------------------
>
> Key: CLOUDSTACK-7424
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7424
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Automation, Test
> Affects Versions: 4.5.0
> Reporter: Chandan Purushothama
> Assignee: Murali Reddy
> Priority: Critical
> Fix For: 4.5.0
>
>
> Error Executing the test case
> *integration.component.test_multiple_ips_per_nic.TestVmNetworkOperations.test_network_restart_cleanup_false_3_VPC*
> *Error Message*
> Job failed: {jobprocstatus : 0, created : u'2014-08-24T07:31:03+0000', cmd :
> u'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd', userid :
> u'77df13d2-2b4d-11e4-89bd-1e5d0e053e75', jobstatus : 2, jobid :
> u'01faacdc-294b-4209-a855-2dc8ea0aa2b4', jobresultcode : 530, jobresulttype :
> u'object', jobresult : {errorcode : 530, errortext : u'Failed to restart
> network'}, accountid : u'77df055e-2b4d-11e4-89bd-1e5d0e053e75'} Logs
> available at
> http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n
> *Stacktrace*
> File "/usr/lib/python2.7/unittest/case.py", line 332, in run
> testMethod()
> File "/usr/local/lib/python2.7/dist-packages/ddt.py", line 114, in wrapper
> return func(self, *args, **kwargs)
> File
> "/root/cloudstack/test/integration/component/test_multiple_ips_per_nic.py",
> line 1186, in test_network_restart_cleanup_false
> network.restart(self.apiclient, cleanup=False)
> File "/usr/local/lib/python2.7/dist-packages/marvin/lib/base.py", line
> 2626, in restart
> return(apiclient.restartNetwork(cmd))
> File
> "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackAPI/cloudstackAPIClient.py",
> line 1899, in restartNetwork
> response = self.connection.marvinRequest(command, response_type=response,
> method=method)
> File
> "/usr/local/lib/python2.7/dist-packages/marvin/cloudstackConnection.py", line
> 379, in marvinRequest
> raise e
> 'Job failed: {jobprocstatus : 0, created : u\'2014-08-24T07:31:03+0000\', cmd
> : u\'org.apache.cloudstack.api.command.user.network.RestartNetworkCmd\',
> userid : u\'77df13d2-2b4d-11e4-89bd-1e5d0e053e75\', jobstatus : 2, jobid :
> u\'01faacdc-294b-4209-a855-2dc8ea0aa2b4\', jobresultcode : 530, jobresulttype
> : u\'object\', jobresult : {errorcode : 530, errortext : u\'Failed to restart
> network\'}, accountid : u\'77df055e-2b4d-11e4-89bd-1e5d0e053e75\'}\n
> Logs available at
> http://xenrt.hq.xensource.com/control/queue.cgi?action=testlogs&id=804076&phase=Parallel&test=multiple_ips_per_n
>
> ==============================
> *Error in the Management Server Log*:
> ==============================
> {code}
> 2014-08-24 07:31:03,602 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is
> in Stopping, so not sending apply static nat rules commands to the backend
> 2014-08-24 07:31:03,621 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is
> in Stopping, so not sending apply vpc ip association commands to the backend
> 2014-08-24 07:31:03,638 DEBUG [c.c.n.r.VirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is
> in Stopping, so not sending apply vpc ip association commands to the backend
> 2014-08-24 07:31:03,640 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Sending network
> shutdown to VpcVirtualRouter
> 2014-08-24 07:31:03,653 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Service
> SecurityGroup is not supported in the network id=314
> 2014-08-24 07:31:03,654 DEBUG [c.c.n.r.VpcVirtualNetworkApplianceManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Router r-229-VM is
> in Stopping, so not sending setup guest network command to the backend
> 2014-08-24 07:31:03,663 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Lock is acquired for
> nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM] from network
> Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,667 DEBUG [c.c.n.NetworkModelImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Service
> SecurityGroup is not supported in the network id=314
> 2014-08-24 07:31:03,668 WARN [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Unable to remove vm
> VM[DomainRouter|r-229-VM] from network Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,669 DEBUG [c.c.v.VirtualMachineManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Lock is released for
> nic id 476 as a part of remove vm VM[DomainRouter|r-229-VM] from network
> Ntwk[314|Guest|37]
> 2014-08-24 07:31:03,671 WARN [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Unable to complete
> shutdown of the network elements due to element: VpcVirtualRouter
> com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is
> unreachable: Unable to remove vm VM[DomainRouter|r-229-VM] from network, is
> not in the right state
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateRemoveVmFromNetwork(VirtualMachineManagerImpl.java:3079)
> at
> com.cloud.vm.VirtualMachineManagerImpl.removeVmFromNetwork(VirtualMachineManagerImpl.java:3009)
> at
> com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl.removeVpcRouterFromGuestNetwork(VpcVirtualNetworkApplianceManagerImpl.java:321)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy190.removeVpcRouterFromGuestNetwork(Unknown Source)
> at
> com.cloud.network.element.VpcVirtualRouterElement.shutdown(VpcVirtualRouterElement.java:261)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.shutdownNetworkElementsAndResources(NetworkOrchestrator.java:2161)
> at
> org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.restartNetwork(NetworkOrchestrator.java:2468)
> at
> com.cloud.network.NetworkServiceImpl.restartNetwork(NetworkServiceImpl.java:1874)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:601)
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
> at
> org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
> at $Proxy157.restartNetwork(Unknown Source)
> at
> org.apache.cloudstack.api.command.user.network.RestartNetworkCmd.execute(RestartNetworkCmd.java:95)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Failed to shutdown
> the network elements and resources as a part of network restart: Implemented
> 2014-08-24 07:31:03,673 DEBUG [o.a.c.e.o.NetworkOrchestrator]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Marking network
> Ntwk[314|Guest|37] with restartRequired=true
> 2014-08-24 07:31:03,675 WARN [c.c.n.NetworkServiceImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529 ctx-3eaba8e6) Network id=314
> failed to restart.
> 2014-08-24 07:31:03,680 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-82:ctx-6738c61c job-1529) Complete async job-1529,
> jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to restart network"}
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
