[
https://issues.apache.org/jira/browse/CLOUDSTACK-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanna Santhanam updated CLOUDSTACK-4315:
-------------------------------------------
Attachment: kvm.tar.bz2
logs (agent + management in DEBUG mode)
> Vpc Network upgrade is failing
> ------------------------------
>
> Key: CLOUDSTACK-4315
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4315
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Affects Versions: 4.2.0
> Reporter: Prasanna Santhanam
> Priority: Critical
> Fix For: 4.2.0
>
> Attachments: kvm.tar.bz2
>
>
> While running tests for vpc network upgrade following situation was
> encountered. Before upgrading the vpc network, the network is shutdown and it
> fails with an NPE
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,732 WARN [cloud.network.NetworkManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ])
> shutdownNetworkRules failed during the netw
> ork Ntwk[441|Guest|127] shutdown due to
> Aug 13 03:17:59 java.lang.NullPointerException
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.createApplyLoadBalancingRulesCommands(VirtualNetworkApplianceManagerImpl.java:3313)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendLBRules(VirtualNetworkApplianceManagerImpl.java:3678)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$9.execute(VirtualNetworkApplianceManagerImpl.java:3671)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3796)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyLoadBalancingRules(VirtualNetworkApplianceManagerImpl.java:3657)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.element.VirtualRouterElement.applyLBRules(VirtualRouterElement.java:393)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:1549)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLbRules(LoadBalancingRulesManagerImpl.java:2089)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.applyLoadBalancerRules(LoadBalancingRulesManagerImpl.java:1586)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.lb.LoadBalancingRulesManagerImpl.revokeLoadBalancersForNetwork(LoadBalancingRulesManagerImpl.java:1519)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.shutdownNetworkResources(NetworkManagerImpl.java:4054)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.shutdownNetworkElementsAndResources(NetworkManagerImpl.java:2913)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.shutdownNetwork(NetworkManagerImpl.java:2859)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1937)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending
> network shutdown to VpcVirtualRoute
> r
> But subsequently the network does shutdown without issues:
> ug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,742 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Sending
> network shutdown to VpcVirtualRouter
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,750 DEBUG [network.element.VpcVirtualRouterElement]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Router
> VM[DomainRouter|r-479-QA] is not a part the network Ntwk[441|Guest|127]
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,750 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Network
> id=441 is shutdown successfully, cleaning up corresponding resources now.
> And then as the upgrade is triggered:
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,766 DEBUG [cloud.network.NetworkManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Lock is
> released for network id 441 as a part of network implement
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,766 WARN [cloud.network.NetworkServiceImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ]) Failed
> to implement network Ntwk[441|Guest|127] elements and resources as a part of
> network update due to
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com
> com.cloud.utils.exception.CloudRuntimeException: Failed to add VPC router
> VM[DomainRouter|r-479-QA] to guest network Ntwk[441|Guest|127]
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.element.VpcVirtualRouterElement.implement(VpcVirtualRouterElement.java:178)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.implementNetworkElementsAndResources(NetworkManagerImpl.java:2003)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkManagerImpl.implementNetwork(NetworkManagerImpl.java:1908)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2240)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,773 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ])
> Unexpected exception while executing
> org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com
> com.cloud.utils.exception.CloudRuntimeException: Failed to implement network
> (with specified id) elements and resources as a part of network update
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2244)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> org.apache.cloudstack.api.command.user.network.UpdateNetworkCmd.execute(UpdateNetworkCmd.java:139)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.FutureTask.run(FutureTask.java:166)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com at
> java.lang.Thread.run(Thread.java:679)
> Aug 13 03:17:59 cloudstack-centos63.fmt.vmops.com local0: 2013-08-13
> 10:17:59,776 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-26:job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ])
> Complete async job-1868 = [ 90c7611d-8239-4976-b0aa-ac7b20d6e230 ],
> jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to
> implement network (with specified id) elements and resources as a part of
> network update
> This is consistenly seen on test runs on kvm environments, but doesn't look to
> be hypervisor specific
> Steps to reproduce:
> # 1. Create a VPC with cidr - 10.1.1.1/16
> # 2. Create a Network offering - NO1 with all supported services
> # except PF services provided by VRVPC provider, conserve mode=OFF
> # 3.Create a Network offering - NO2 with all supported services
> # including Pf services provided by VRVPC provider,conserve mode=OFF
> # 4. Add network1(10.1.1.1/24) using N01 to this VPC.
> # 5. Deploy vm1 and vm2 in network1.
> # 6. Create a Static Nat and LB rules for vms in network1.
> # 7. Make sure you are not allowed to create a PF rule for any Vm in
> # network1 and the Static Nat and LB rules for vms work as expected
> # 8. Update network1 to NO2.
> Management server logs are attached.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
