[ https://issues.apache.org/jira/browse/CLOUDSTACK-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Murali Reddy updated CLOUDSTACK-6796: ------------------------------------- Assignee: (was: Murali Reddy) > [OVS]Failure in network update does not change network offering to original > offering > ------------------------------------------------------------------------------------ > > Key: CLOUDSTACK-6796 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6796 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server, Network Controller > Affects Versions: 4.4.0 > Environment: Latest build from 4.4 with commit > d130530bd3e1cd6d8249d5045e00e4e4e2201521 > Reporter: Sanjeev N > Priority: Critical > Labels: ovs > Fix For: Future > > Attachments: management-server.rar > > > [OVS]Failure in network update does not change network offering to original > offering hence starting vms would fail in the network > Steps to Reproduce: > =============== > 1.Bring up CS in advanced zone with xen cluster > 2.Create physical network with GRE isolation > 3.Create network with default offering > "DefaultIsolatedNetworkOfferingWithSourceNatService" > 4.Deploy few vms in the above netwrok > 5.Create another network offering with virtual networking service and OVS as > the connectivity service provider > 6.Stop all the vms in the network > 7.Update network with new offering created at step5 > Results: > ====== > Network update will fail from vlan isolation to connectivity service due to > bug CS-6795. However the network offering id for the network is changing to > new network offering. It is not setting back to default isolated network > offering. > mysql> select * from ntwk_offering_service_map where network_offering_id=15; > +----+---------------------+----------------+---------------+---------------------+ > | id | network_offering_id | service | provider | created > | > +----+---------------------+----------------+---------------+---------------------+ > | 60 | 15 | Connectivity | Ovs | 2014-05-26 > 12:51:34 | > | 55 | 15 | Dhcp | VirtualRouter | 2014-05-26 > 12:51:34 | > | 54 | 15 | Dns | VirtualRouter | 2014-05-26 > 12:51:34 | > | 61 | 15 | Firewall | VirtualRouter | 2014-05-26 > 12:51:34 | > | 58 | 15 | Lb | VirtualRouter | 2014-05-26 > 12:51:34 | > | 57 | 15 | PortForwarding | VirtualRouter | 2014-05-26 > 12:51:34 | > | 56 | 15 | SourceNat | VirtualRouter | 2014-05-26 > 12:51:34 | > | 59 | 15 | StaticNat | VirtualRouter | 2014-05-26 > 12:51:34 | > | 53 | 15 | UserData | VirtualRouter | 2014-05-26 > 12:51:34 | > +----+---------------------+----------------+---------------+---------------------+ > 9 rows in set (0.00 sec) > Following is the network created with default isolated network offering but > after network update failure the offering still shows the new offering: > mysql> select * from networks where id=211\G; > *************************** 1. row *************************** > id: 211 > name: vlan1 > uuid: f803e17f-b59b-4229-9e70-5bb4fcfc2570 > display_text: vlan1 > traffic_type: Guest > broadcast_domain_type: Vlan > broadcast_uri: vlan://986 > gateway: 10.1.1.1 > cidr: 10.1.1.0/24 > mode: Dhcp > network_offering_id: 15 > physical_network_id: 200 > data_center_id: 1 > guru_name: ExternalGuestNetworkGuru > state: Shutdown > related: 211 > domain_id: 1 > account_id: 2 > dns1: NULL > dns2: NULL > guru_data: NULL > set_fields: 0 > acl_type: Account > network_domain: cs2cloud.internal > reservation_id: c2b3cb64-adfd-4722-9aed-8d2d7710e32f > guest_type: Isolated > restart_required: 0 > created: 2014-05-28 11:09:16 > removed: NULL > specify_ip_ranges: 0 > vpc_id: NULL > ip6_gateway: NULL > ip6_cidr: NULL > network_cidr: NULL > display_network: 1 > network_acl_id: NULL > streched_l2: 0 > 1 row in set (0.00 sec) > ERROR: > No query specified > Impact of this: > =========== > Since the network offering is with connectivity service , CS is failed to > implement the network and vm start is failing. > 2014-05-28 07:52:28,188 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if > OvsElement can handle service SourceNat on network vlan1 > 2014-05-28 07:52:28,189 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual > router element doesn't need to associate ip addresses on the backend; virtual > router doesn't exist in the network 211 > 2014-05-28 07:52:28,193 DEBUG [c.c.n.e.VirtualRouterElement] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual > router elemnt doesn't need to associate ip addresses on the backend; virtual > router doesn't exist in the network 211 > 2014-05-28 07:52:28,195 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending > network shutdown to Ovs > 2014-05-28 07:52:28,195 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if > OvsElement can handle service Connectivity on network vlan1 > 2014-05-28 07:52:28,195 WARN [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Unable to > complete shutdown of the network elements due to element: Ovs > 2014-05-28 07:52:28,197 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending > network shutdown to VirtualRouter > 2014-05-28 07:52:28,201 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is > released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a > part of network shutdown > 2014-05-28 07:52:28,202 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is > released for network id 211 as a part of network implement > 2014-05-28 07:52:28,202 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Failed to > start instance VM[User|i-2-25-VM] > java.lang.NullPointerException > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180) > at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-28 07:52:28,206 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Cleaning up > resources for the vm VM[User|i-2-25-VM] in Starting state > 2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq > 1-8670555182595055942: Sending { Cmd , MgmtId: 7332683579487, via: > 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] > } > 2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq > 1-8670555182595055942: Executing: { Cmd , MgmtId: 7332683579487, via: > 1(Rack1Pod1Host13), Ver: v1, Flags: 100011, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}] > } > 2014-05-28 07:52:28,209 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Executing request > 2014-05-28 07:52:28,215 INFO [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-304:ctx-21497aaa) VM does not exist on > XenServer693592c3-3a6d-4025-84f2-64c52b733568 > 2014-05-28 07:52:28,215 DEBUG [c.c.a.m.DirectAgentAttache] > (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Response Received: > 2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] > (DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Processing: { Ans: > , MgmtId: 7332683579487, via: 1, Ver: v1, Flags: 10, > [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not > exist","wait":0}}] } > 2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq > 1-8670555182595055942: Received: { Ans: , MgmtId: 7332683579487, via: 1, > Ver: v1, Flags: 10, { StopAnswer } } > 2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully > released network resources for the vm VM[User|i-2-25-VM] > 2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully > cleanued up resources for the vm VM[User|i-2-25-VM] in Starting state > 2014-05-28 07:52:28,229 DEBUG [c.c.c.CapacityManagerImpl] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) VM state > transitted from :Starting to Stopped with event: OperationFailedvm's original > host id: 1 new host id: null host id before state transition: 1 > 2014-05-28 07:52:28,248 INFO [c.c.v.VmWorkJobHandlerProxy] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Rethrow > exception com.cloud.exception.AgentUnavailableException: Resource [Host:1] is > unreachable: Host 1: Unable to start instance due to null > 2014-05-28 07:52:28,248 DEBUG [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Done with run of VM work > job: com.cloud.vm.VmWorkStart for VM 25, job origin: 122 > 2014-05-28 07:52:28,248 ERROR [c.c.v.VmWorkJobDispatcher] > (Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Unable to complete > AsyncJobVO {id:123, userId: 2, accountId: 2, instanceType: null, instanceId: > null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: > rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA, > cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: > null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null, > lastPolled: null, created: Wed May 28 07:52:26 EDT 2014}, job origin:122 > com.cloud.exception.AgentUnavailableException: Resource [Host:1] is > unreachable: Host 1: Unable to start instance due to null > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1103) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180) > at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > Caused by: java.lang.NullPointerException > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983) > ... 19 more > 014-05-28 07:52:30,312 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if > OvsElement can handle service SourceNat on network vlan1 > 2014-05-28 07:52:30,314 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual > router element doesn't need to associate ip addresses on the backend; virtual > router doesn't exist in the network 211 > 2014-05-28 07:52:30,317 DEBUG [c.c.n.e.VirtualRouterElement] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual > router elemnt doesn't need to associate ip addresses on the backend; virtual > router doesn't exist in the network 211 > 2014-05-28 07:52:30,319 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending > network shutdown to Ovs > 2014-05-28 07:52:30,320 DEBUG [c.c.n.e.OvsElement] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if > OvsElement can handle service Connectivity on network vlan1 > 2014-05-28 07:52:30,320 WARN [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Unable to > complete shutdown of the network elements due to element: Ovs > 2014-05-28 07:52:30,322 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending > network shutdown to VirtualRouter > 2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is > released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a > part of network shutdown > 2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is > released for network id 211 as a part of network implement > 2014-05-28 07:52:30,326 ERROR [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Failed to > start instance VM[User|i-2-25-VM] > java.lang.NullPointerException > at > org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983) > at > com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180) > at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) > at > com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325) > at > com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > 2014-05-28 07:52:30,331 DEBUG [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Cleaning up > resources for the vm VM[User|i-2-25-VM] in Starting state -- This message was sent by Atlassian JIRA (v6.3.4#6332)