Sanjeev N created CLOUDSTACK-6796:
-------------------------------------
Summary: [OVS]Failure in network update does not change network
offering to original offering
Key: CLOUDSTACK-6796
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6796
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server, Network Controller
Affects Versions: 4.4.0
Environment: Latest build from 4.4 with commit
d130530bd3e1cd6d8249d5045e00e4e4e2201521
Reporter: Sanjeev N
Assignee: Murali Reddy
Priority: Critical
Fix For: 4.4.0
Attachments: management-server.rar
[OVS]Failure in network update does not change network offering to original
offering hence starting vms would fail in the network
Steps to Reproduce:
===============
1.Bring up CS in advanced zone with xen cluster
2.Create physical network with GRE isolation
3.Create network with default offering
"DefaultIsolatedNetworkOfferingWithSourceNatService"
4.Deploy few vms in the above netwrok
5.Create another network offering with virtual networking service and OVS as
the connectivity service provider
6.Stop all the vms in the network
7.Update network with new offering created at step5
Results:
======
Network update will fail from vlan isolation to connectivity service due to bug
CS-6795. However the network offering id for the network is changing to new
network offering. It is not setting back to default isolated network offering.
mysql> select * from ntwk_offering_service_map where network_offering_id=15;
+----+---------------------+----------------+---------------+---------------------+
| id | network_offering_id | service | provider | created
|
+----+---------------------+----------------+---------------+---------------------+
| 60 | 15 | Connectivity | Ovs | 2014-05-26
12:51:34 |
| 55 | 15 | Dhcp | VirtualRouter | 2014-05-26
12:51:34 |
| 54 | 15 | Dns | VirtualRouter | 2014-05-26
12:51:34 |
| 61 | 15 | Firewall | VirtualRouter | 2014-05-26
12:51:34 |
| 58 | 15 | Lb | VirtualRouter | 2014-05-26
12:51:34 |
| 57 | 15 | PortForwarding | VirtualRouter | 2014-05-26
12:51:34 |
| 56 | 15 | SourceNat | VirtualRouter | 2014-05-26
12:51:34 |
| 59 | 15 | StaticNat | VirtualRouter | 2014-05-26
12:51:34 |
| 53 | 15 | UserData | VirtualRouter | 2014-05-26
12:51:34 |
+----+---------------------+----------------+---------------+---------------------+
9 rows in set (0.00 sec)
Following is the network created with default isolated network offering but
after network update failure the offering still shows the new offering:
mysql> select * from networks where id=211\G;
*************************** 1. row ***************************
id: 211
name: vlan1
uuid: f803e17f-b59b-4229-9e70-5bb4fcfc2570
display_text: vlan1
traffic_type: Guest
broadcast_domain_type: Vlan
broadcast_uri: vlan://986
gateway: 10.1.1.1
cidr: 10.1.1.0/24
mode: Dhcp
network_offering_id: 15
physical_network_id: 200
data_center_id: 1
guru_name: ExternalGuestNetworkGuru
state: Shutdown
related: 211
domain_id: 1
account_id: 2
dns1: NULL
dns2: NULL
guru_data: NULL
set_fields: 0
acl_type: Account
network_domain: cs2cloud.internal
reservation_id: c2b3cb64-adfd-4722-9aed-8d2d7710e32f
guest_type: Isolated
restart_required: 0
created: 2014-05-28 11:09:16
removed: NULL
specify_ip_ranges: 0
vpc_id: NULL
ip6_gateway: NULL
ip6_cidr: NULL
network_cidr: NULL
display_network: 1
network_acl_id: NULL
streched_l2: 0
1 row in set (0.00 sec)
ERROR:
No query specified
Impact of this:
===========
Since the network offering is with connectivity service , CS is failed to
implement the network and vm start is failing.
2014-05-28 07:52:28,188 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if
OvsElement can handle service SourceNat on network vlan1
2014-05-28 07:52:28,189 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router
element doesn't need to associate ip addresses on the backend; virtual router
doesn't exist in the network 211
2014-05-28 07:52:28,193 DEBUG [c.c.n.e.VirtualRouterElement]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Virtual router
elemnt doesn't need to associate ip addresses on the backend; virtual router
doesn't exist in the network 211
2014-05-28 07:52:28,195 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending
network shutdown to Ovs
2014-05-28 07:52:28,195 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Checking if
OvsElement can handle service Connectivity on network vlan1
2014-05-28 07:52:28,195 WARN [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Unable to
complete shutdown of the network elements due to element: Ovs
2014-05-28 07:52:28,197 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Sending
network shutdown to VirtualRouter
2014-05-28 07:52:28,201 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is
released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a
part of network shutdown
2014-05-28 07:52:28,202 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Lock is
released for network id 211 as a part of network implement
2014-05-28 07:52:28,202 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Failed to
start instance VM[User|i-2-25-VM]
java.lang.NullPointerException
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-28 07:52:28,206 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Cleaning up
resources for the vm VM[User|i-2-25-VM] in Starting state
2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq
1-8670555182595055942: Sending { Cmd , MgmtId: 7332683579487, via:
1(Rack1Pod1Host13), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}]
}
2014-05-28 07:52:28,209 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq
1-8670555182595055942: Executing: { Cmd , MgmtId: 7332683579487, via:
1(Rack1Pod1Host13), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-2-25-VM","wait":0}}]
}
2014-05-28 07:52:28,209 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Executing request
2014-05-28 07:52:28,215 INFO [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-304:ctx-21497aaa) VM does not exist on
XenServer693592c3-3a6d-4025-84f2-64c52b733568
2014-05-28 07:52:28,215 DEBUG [c.c.a.m.DirectAgentAttache]
(DirectAgent-304:ctx-21497aaa) Seq 1-8670555182595055942: Response Received:
2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request] (DirectAgent-304:ctx-21497aaa)
Seq 1-8670555182595055942: Processing: { Ans: , MgmtId: 7332683579487, via: 1,
Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not
exist","wait":0}}] }
2014-05-28 07:52:28,215 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Seq
1-8670555182595055942: Received: { Ans: , MgmtId: 7332683579487, via: 1, Ver:
v1, Flags: 10, { StopAnswer } }
2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully
released network resources for the vm VM[User|i-2-25-VM]
2014-05-28 07:52:28,221 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Successfully
cleanued up resources for the vm VM[User|i-2-25-VM] in Starting state
2014-05-28 07:52:28,229 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) VM state
transitted from :Starting to Stopped with event: OperationFailedvm's original
host id: 1 new host id: null host id before state transition: 1
2014-05-28 07:52:28,248 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123 ctx-ccbcca96) Rethrow
exception com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
2014-05-28 07:52:28,248 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Done with run of VM work
job: com.cloud.vm.VmWorkStart for VM 25, job origin: 122
2014-05-28 07:52:28,248 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-47:ctx-3e2b5a9e job-122/job-123) Unable to complete
AsyncJobVO {id:123, userId: 2, accountId: 2, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkStart, cmdInfo:
rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIAC0oABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAdwbGFubmVycQB-AANMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAGXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcHEAfgAKcHNyABFqYXZhLnV0aWwuSGFzaE1hcAUH2sHDFmDRAwACRgAKbG9hZEZhY3RvckkACXRocmVzaG9sZHhwP0AAAAAAAAx3CAAAABAAAAABdAAKVm1QYXNzd29yZHQAHHJPMEFCWFFBRG5OaGRtVmtYM0JoYzNOM2IzSmt4cA,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 7332683579487, completeMsid: null, lastUpdated: null,
lastPolled: null, created: Wed May 28 07:52:26 EDT 2014}, job origin:122
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
unreachable: Host 1: Unable to start instance due to null
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1103)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NullPointerException
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
... 19 more
014-05-28 07:52:30,312 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if
OvsElement can handle service SourceNat on network vlan1
2014-05-28 07:52:30,314 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router
element doesn't need to associate ip addresses on the backend; virtual router
doesn't exist in the network 211
2014-05-28 07:52:30,317 DEBUG [c.c.n.e.VirtualRouterElement]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Virtual router
elemnt doesn't need to associate ip addresses on the backend; virtual router
doesn't exist in the network 211
2014-05-28 07:52:30,319 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending
network shutdown to Ovs
2014-05-28 07:52:30,320 DEBUG [c.c.n.e.OvsElement]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Checking if
OvsElement can handle service Connectivity on network vlan1
2014-05-28 07:52:30,320 WARN [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Unable to
complete shutdown of the network elements due to element: Ovs
2014-05-28 07:52:30,322 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Sending
network shutdown to VirtualRouter
2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is
released for network Ntwk[f803e17f-b59b-4229-9e70-5bb4fcfc2570|Guest|15] as a
part of network shutdown
2014-05-28 07:52:30,326 DEBUG [o.a.c.e.o.NetworkOrchestrator]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Lock is
released for network id 211 as a part of network implement
2014-05-28 07:52:30,326 ERROR [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Failed to
start instance VM[User|i-2-25-VM]
java.lang.NullPointerException
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1273)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:983)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5180)
at sun.reflect.GeneratedMethodAccessor436.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5325)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-05-28 07:52:30,331 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-48:ctx-56a17636 job-122/job-124 ctx-bd415b93) Cleaning up
resources for the vm VM[User|i-2-25-VM] in Starting state
--
This message was sent by Atlassian JIRA
(v6.2#6252)
