[
https://issues.apache.org/jira/browse/CLOUDSTACK-4616?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangeetha Hariharan closed CLOUDSTACK-4616.
-------------------------------------------
Tested the following use case which resulted in host being marked as "Up" ,
even though the Primary store was down resulting in system Vms being constantly
attempting to start and fail subsequently.
Tested with latest build from 4.3:
1. Deploy few Vms in each of the hosts , so we start with 10 Vms.
2. Create snaposhot for ROOT volumes.
3. When snapshot is still in progress , Make the primary storage unavailable
for 10 mts.
This results in the KVM hosts to reboot.
But reboot of KVM host is not successful.It is stuck at trying to unmount nfs
mount points. This is tracked in CLOUDSTACK-5429.
Stop and start KM hosts manually to workaround this problem.
KVM is now marked as being in "Up" state , even though the Primary store is
down and the agent was not able to create a mount point.
At this point all the Vms are marked as "Stopped" state in CloudStack.
System Vms being constantly attempting to start and fail subsequently.
When system Vm fails to start , we see that the link local Ip address gets
release back to the pool.
mysql> select * from op_dc_link_local_ip_address_alloc where data_center_id=1
and taken is not null;
The above query did not return Ip address as being allocated for SSVM which m
failed to start many times.
4. Now make the primary store available.
5. Force reconnect the host.
After this we see the System Vm started successfully.
> When system Vms fail to start when host is down , link local Ip addresses do
> not get released resulting in all the link local Ip addresses being consumed
> eventually.
> ----------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-4616
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-4616
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.1
> Environment: Build from 4.2-forward
> Reporter: Sangeetha Hariharan
> Assignee: Murali Reddy
> Priority: Critical
> Fix For: 4.3.0
>
> Attachments: hostdown.rar
>
>
> When system Vms fail to start when host is down , link local Ip addresses do
> not get released resulting in all the link local Ip addresses being consumed
> eventually.
> Steps to reproduce the problem:
> Advanced zone with 1 cluster having 1 host (Xenserver).
> Had SSVM,CCPVM, 2 routers and few user Vms running in the host.
> power down the host.
> When host was powered down , host is still marked as being in "Up" state .
> Bug tracked in - CLOUDSTACK-2140.
> Attempt to restart all the system Vms in the host that is down is made
> continuously and it fails.
> These failed attempts do not result in releasing the linked local Ip ,
> resulting in all linked local Ips being consumed.
> When the host is actually powered on , attempts to start the System Vms fail
> , because of teh following exception seen in the management-server.logs:
> 013-09-05 12:00:09,551 INFO [cloud.vm.VirtualMachineManagerImpl]
> (secstorage-1:null) Insufficient capacity
> com.cloud.exception.InsufficientAddressCapacityException: Insufficient link
> local address capacityScope=interface com.cloud.dc.DataCenter; id=1
> at
> com.cloud.network.guru.ControlNetworkGuru.reserve(ControlNetworkGuru.java:156)
> at
> com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2157)
> at
> com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2127)
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:578)
> at
> com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:571)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
> at
> com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
> at
> com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
> at
> com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
> at
> com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
> at
> com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
> at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> mysql> select * from op_dc_link_local_ip_address_alloc where data_center_id=1
> and taken is null;
> Empty set (0.00 sec)
>
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
