Sangeetha Hariharan created CLOUDSTACK-7742:
-----------------------------------------------
Summary: Xenserver HA - SSVM failing to start since it is running
out of management ip address
Key: CLOUDSTACK-7742
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7742
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.5.0
Environment: Build from master
Reporter: Sangeetha Hariharan
HA - SSVM failing to start since it is running out of management ip address
Set up:
Cluster with 3 Xenserver hosts.
I am executing host HA scenarios where host is being brought down ( or
simulating contol path network failure / storage network failure).
After couple of such scenarios , i see that the SSVM fails to start as part of
HA the reason being running out of management nic:
management server logs:
014-10-16 12:15:44,311 DEBUG [c.c.u.d.T.Transaction]
(Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6) Rolling back
the transaction: Time = 1 Name = Work-Job-Executor-106; called by
-TransactionLegacy.rollback:902-DataCenterIpAddressDaoImpl.takeIpAddress:61-GeneratedMethodAccessor493.invoke:-1-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopUtils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150-TransactionContextInterceptor.invoke:34-ReflectiveMethodInvocation.proceed:161-ExposeInvocationInterceptor.invoke:91-ReflectiveMethodInvocation.proceed:172
2014-10-16 12:15:44,312 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6) Insufficient
capacity
com.cloud.exception.InsufficientAddressCapacityException: Unable to get a
management ip addressScope=interface com.cloud.dc.Pod; id=1
at
com.cloud.network.guru.PodBasedNetworkGuru.reserve(PodBasedNetworkGuru.java:123)
at
com.cloud.network.guru.StorageNetworkGuru.reserve(StorageNetworkGuru.java:122)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepareNic(NetworkOrchestrator.java:1338)
at
org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.prepare(NetworkOrchestrator.java:1309)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:970)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4590)
at sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4746)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:513)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:470)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2014-10-16 12:15:44,324 DEBUG [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-106:ctx-323991ca job-771/job-943 ctx-3a2e9ed6) Cleaning up
resources for the vm VM[SecondaryStorageVm|s-115-VM] in Starting state
There are 2 issues here:
1. Some of the SSVMs that are in destroyed state still have not released the
management Ips back to the freepool.
2. Some of these destroyed SSVMs have 2 management ip addresses associated with
it . why is this the case?
3. I still see 1 management ip address that is free , but SSVM is still not
able to come up.
mysql> select id,name,state from vm_instance where id in (1,7,18,71);
+----+---------+-----------+
| id | name | state |
+----+---------+-----------+
| 1 | v-1-VM | Running |
| 7 | s-7-VM | Destroyed |
| 18 | s-18-VM | Destroyed |
| 71 | s-71-VM | Destroyed |
+----+---------+-----------+
4 rows in set (0.00 sec)
mysql> select instance_id from nics where id in (select nic_id from
op_dc_ip_address_alloc where taken is not null);
+-------------+
| instance_id |
+-------------+
| 1 |
| 7 |
| 7 |
| 18 |
| 18 |
| 71 |
| 71 |
+-------------+
7 rows in set (0.00 sec)
mysql> select * from op_dc_ip_address_alloc ;
+----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
| id | ip_address | data_center_id | pod_id | nic_id | reservation_id
| taken | mac_address |
+----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
| 1 | 10.223.59.69 | 1 | 1 | 261 |
5cf7268d-cf6c-421d-86d7-fe124b055bd5 | 2014-10-15 00:20:44 | 1 |
| 2 | 10.223.59.70 | 1 | 1 | 3 |
a5254254-b384-4783-adab-05bb7d6355be | 2014-10-16 17:34:42 | 2 |
| 3 | 10.223.59.71 | 1 | 1 | 261 |
a89646ab-ce75-4518-b685-5fa49e98faca | 2014-10-13 22:05:14 | 3 |
| 4 | 10.223.59.72 | 1 | 1 | 26 |
4c197653-f794-438b-b743-f80cc7348be0 | 2014-10-10 18:47:44 | 4 |
| 5 | 10.223.59.73 | 1 | 1 | 3 |
2c15e101-b534-4b4e-bebb-f580ea284c01 | 2014-10-16 18:31:42 | 5 |
| 6 | 10.223.59.74 | 1 | 1 | 27 |
4c197653-f794-438b-b743-f80cc7348be0 | 2014-10-10 18:47:44 | 6 |
| 7 | 10.223.59.75 | 1 | 1 | 49 |
680321c2-9290-4915-aec4-1b37dc5aefbd | 2014-10-11 00:24:44 | 7 |
| 8 | 10.223.59.76 | 1 | 1 | 48 |
680321c2-9290-4915-aec4-1b37dc5aefbd | 2014-10-11 00:24:44 | 8 |
| 9 | 10.223.59.77 | 1 | 1 | NULL | NULL
| NULL | 9 |
| 10 | 10.223.59.78 | 1 | 1 | 3 |
9980f72c-4bc8-40ed-87bf-99fd375bd13d | 2014-10-10 18:48:12 | 10 |
| 11 | 10.223.59.79 | 1 | 1 | 260 |
5cf7268d-cf6c-421d-86d7-fe124b055bd5 | 2014-10-15 00:20:44 | 11 |
| 12 | 10.223.59.80 | 1 | 1 | 260 |
a89646ab-ce75-4518-b685-5fa49e98faca | 2014-10-13 22:05:14 | 12 |
+----+--------------+----------------+--------+--------+--------------------------------------+---------------------+-------------+
12 rows in set (0.00 sec)
mysql>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
