Prasanna Santhanam created CLOUDSTACK-4315:
----------------------------------------------
Summary: If network shutdown fails, subsequent network upgrade
appears to be failing
Key: CLOUDSTACK-4315
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4315
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Reporter: Prasanna Santhanam
Priority: Critical
While running tests for vpc network upgrade following situation was
encountered. Before upgrading the vpc network, the network is shutdown and it
fails with an NPE
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,732 WARN [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) shutdownNetworkRules failed during
the netw
ork Ntwk[441|Guest|127] shutdown due to
Aug 13 03:17:59 java.lang.NullPointerException
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createApplyLoadBalancingRulesCommands(VirtualNetworkApplianceManagerImpl.java:3313)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendLBRules(VirtualNetworkApplianceManagerImpl.java:3678)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl$9.execute(VirtualNetworkApplianceManagerImpl.java:3671)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3657)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:393)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1549)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2089)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1586)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.lb.LoadBalancingRulesManagerImpl.revokeLoadBalancersForNetwork(LoadBalancingRulesManagerImpl.java:1519)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.shutdownNetworkResources(NetworkManagerImpl.java:4054)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.shutdownNetworkElementsAndResources(NetworkManagerImpl.java:2913)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.shutdownNetwork(NetworkManagerImpl.java:2859)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1937)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask.run(FutureTask.java:166)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.lang.Thread.run(Thread.java:679)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending network shutdown to
VpcVirtualRoute
r
But subsequently the network does shutdown without issues:
ug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending network shutdown to
VpcVirtualRouter
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,750 DEBUG [network.element.VpcVirtualRouterElement]
(Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Router
VM[DomainRouter|r-479-QA] is not a part the network Ntwk[441|Guest|127]
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,750 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Network id=441 is shutdown
successfully, cleaning up corresponding resources now.
And then as the upgrade is triggered:
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,766 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Lock is released for network id 441
as a part of network implement
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,766 WARN [cloud.network.NetworkServiceImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Failed to implement network
Ntwk[441|Guest|127] elements and resources as a part of network update due to
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com
com.cloud.utils.exception.CloudRuntimeException: Failed to add VPC router
VM[DomainRouter|r-479-QA] to guest network Ntwk[441|Guest|127]
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:178)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2003)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1908)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask.run(FutureTask.java:166)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.lang.Thread.run(Thread.java:679)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,773 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Unexpected exception while
executing org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com
com.cloud.utils.exception.CloudRuntimeException: Failed to implement network
(with specified id) elements and resources as a part of network update
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2244)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.FutureTask.run(FutureTask.java:166)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
java.lang.Thread.run(Thread.java:679)
Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
10:17:59,776 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-26:job-1868
= [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Complete async job-1868 = [
90c7611d-8239-4976-b0aa-ac7b20d6e230 ], jobStatus: 2, resultCode: 530, result:
Error Code: 530 Error text: Failed to implement network (with specified id)
elements and resources as a part of network update
This is consistenly seen on test runs on kvm environments, but doesn't look to
be hypervisor specific
Steps to reproduce:
# 1. Create a VPC with cidr - 10.1.1.1/16
# 2. Create a Network offering - NO1 with all supported services
# except PF services provided by VRVPC provider, conserve mode=OFF
# 3.Create a Network offering - NO2 with all supported services
# including Pf services provided by VRVPC provider,conserve mode=OFF
# 4. Add network1(10.1.1.1/24) using N01 to this VPC.
# 5. Deploy vm1 and vm2 in network1.
# 6. Create a Static Nat and LB rules for vms in network1.
# 7. Make sure you are not allowed to create a PF rule for any Vm in
# network1 and the Static Nat and LB rules for vms work as expected
# 8. Update network1 to NO2.
Management server logs are attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
