Sangeetha Hariharan created CLOUDSTACK-4616:
-----------------------------------------------
Summary: When system Vms fail to start when host is down , link
local Ip addresses do not get released resulting in all the link local Ip
addresses being consumed eventually.
Key: CLOUDSTACK-4616
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4616
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.1
Environment: Build from 4.2-forward
Reporter: Sangeetha Hariharan
Priority: Critical
Fix For: 4.2.1
When system Vms fail to start when host is down , link local Ip addresses do
not get released resulting in all the link local Ip addresses being consumed
eventually.
Steps to reproduce the problem:
Advanced zone with 1 cluster having 1 host (Xenserver).
Had SSVM,CCPVM, 2 routers and few user Vms running in the host.
power down the host.
When host was powered down , host is still marked as being in "Up" state . Bug
tracked in - CLOUDSTACK-2140.
Attempt to restart all the system Vms in the host that is down is made
continuously and it fails.
These failed attempts do not result in releasing the linked local Ip ,
resulting in all linked local Ips being consumed.
When the host is actually powered on , attempts to start the System Vms fail ,
because of teh following exception seen in the management-server.logs:
013-09-05 12:00:09,551 INFO [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Insufficient capacity
com.cloud.exception.InsufficientAddressCapacityException: Insufficient link
local address capacityScope=interface com.cloud.dc.DataCenter; id=1
at
com.cloud.network.guru.ControlNetworkGuru.reserve(ControlNetworkGuru.java:156)
at
com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2157)
at
com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
mysql> select * from op_dc_link_local_ip_address_alloc where data_center_id=1
and taken is null;
Empty set (0.00 sec)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
