[
https://issues.apache.org/jira/browse/CLOUDSTACK-5401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
manasaveloori reopened CLOUDSTACK-5401:
---------------------------------------
Steps:
1. Created a cluster with 2 hosts( xen 6.0.2)
2. Created 8VMs in host1 and 4 VMs in host2.
3. Now changed the pool.storage.capacity.disablethreshold to 0.1
4. Kept host 1 in maintenance.
Observation.
Few VMs got migrated successfully.Observing the following message in logs while
migrating some VMs:
2014-01-28 22:24:00,358 DEBUG [o.a.c.s.a.AbstractStoragePoolAllocator]
(secstorage-1:ctx-6c023d59) Checking if storage pool is suitable, name: null
,poolId: 1
2014-01-28 22:24:00,365 DEBUG [c.c.s.StorageManagerImpl]
(secstorage-1:ctx-6c023d59) Checking pool 1 for storage, totalSize:
5902284816384, usedBytes: 3683417358336, usedPct: 0.6240663527641528, disable
threshold: 0.1
2014-01-28 22:24:00,365 DEBUG [c.c.s.StorageManagerImpl]
(secstorage-1:ctx-6c023d59) Insufficient space on pool: 1 since its usage
percentage: 0.6240663527641528 has crossed the
pool.storage.capacity.disablethreshold: 0.1
2014-01-28 22:24:00,365 DEBUG [o.a.c.s.a.ClusterScopeStoragePoolAllocator]
(secstorage-1:ctx-6c023d59) ClusterScopeStoragePoolAllocator returning 0
suitable storage pools
2014-01-28 22:24:00,365 DEBUG [o.a.c.s.a.ZoneWideStoragePoolAllocator]
(secstorage-1:ctx-6c023d59) ZoneWideStoragePoolAllocator to find storage pool
2014-01-28 22:24:00,369 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:ctx-6c023d59) No suitable pools found for volume:
Vol[18|vm=18|ROOT] under cluster: 1
2014-01-28 22:24:00,369 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:ctx-6c023d59) No suitable pools found
2014-01-28 22:24:00,369 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:ctx-6c023d59) No suitable storagePools found under this Cluster: 1
2014-01-28 22:24:00,374 DEBUG [c.c.d.DeploymentPlanningManagerImpl]
(secstorage-1:ctx-6c023d59) Could not find suitable Deployment Destination for
this VM under any clusters, returning.
2014-01-28 22:24:00,375 DEBUG [c.c.d.FirstFitPlanner]
(secstorage-1:ctx-6c023d59) Searching all possible resources under this Zone: 1
2014-01-28 22:24:00,376 DEBUG [c.c.d.FirstFitPlanner]
(secstorage-1:ctx-6c023d59) Listing clusters in order of aggregate capacity,
that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2014-01-28 22:24:00,380 DEBUG [c.c.d.FirstFitPlanner]
(secstorage-1:ctx-6c023d59) Removing from the clusterId list these clusters
from avoid set: [1]
2014-01-28 22:24:00,380 DEBUG [c.c.d.FirstFitPlanner]
(secstorage-1:ctx-6c023d59) No clusters found after removing disabled clusters
and clusters in avoid list, returning.
2014-01-28 22:24:00,398 DEBUG [c.c.c.CapacityManagerImpl]
(secstorage-1:ctx-6c023d59) VM state transitted from :Starting to Stopped with
event: OperationFailedvm's original host id: null new host id: null host id
before state transition: null
2014-01-28 22:24:00,401 WARN [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-6c023d59) Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-18-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:921)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:761)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:745)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:261)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:693)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1270)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:35)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:88)
at
com.cloud.vm.SystemVmLoadScanner$1.runInContext(SystemVmLoadScanner.java:79)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2014-01-28 22:24:00,404 INFO [c.c.s.s.SecondaryStorageManagerImpl]
(secstorage-1:ctx-6c023d59) Unable to start secondary storage vm for standby
capacity, secStorageVm vm Id : 18, will recycle it and start a new one
Threshold check is still done during pool allocation.
Attaching the Ms logs.
> VM migration during host maintenance fails if
> pool.storage.capacity.disablethreshold is lowered
> -----------------------------------------------------------------------------------------------
>
> Key: CLOUDSTACK-5401
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5401
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.1, 4.3.0
> Reporter: Prachi Damle
> Assignee: Prachi Damle
> Priority: Critical
> Fix For: 4.3.0
>
>
> 1. Create a 2 host XS 6.0.2 cluster (H1 and H2)
> 2. Create 6 or more VMs such that they get created in H1
> 3. Lower pool.storage.capacity.disablethreshold to 0.1 (default is 0.85)
> 4. Put H1 into maintenance. Some or all guest VMs fail to migrate to H2
> 2013-11-25 15:41:12,098 DEBUG [cloud.deploy.FirstFitPlanner]
> (HA-Worker-3:work-28) The specified cluster is in avoid set, returning.
> 2013-11-25 15:41:12,098 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (HA-Worker-3:work-28) Unable to find destination for migrating the vm
> VM[User|z1V6]
> 2013-11-25 15:41:12,098 WARN [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-3:work-28) Insufficient capacity for migrating a VM.
> 2013-11-25 15:41:12,099 DEBUG [cloud.resource.ResourceManagerImpl]
> (HA-Worker-3:work-28) No next resource state for host 5 while current state
> is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for
> current state =ErrorInMaintenance event =UnableToMigrate
> at
> com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at
> com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at
> com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
> at
> com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-11-25 15:41:12,100 INFO [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-3:work-28) Rescheduling HAWork[28-Migration-9-Running-Migrating]
> to try again at Mon Nov 25 15:43:14 PST 2013
> 2013-11-25 15:41:12,100 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) Checking suitable pools for volume (Id, Type): (13,ROOT)
> 2013-11-25 15:41:12,100 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) We need to allocate new storagepool for this volume
> 2013-11-25 15:41:12,102 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) Calling StoragePoolAllocators to find suitable pools
> 2013-11-25 15:41:12,103 DEBUG [storage.allocator.LocalStoragePoolAllocator]
> (HA-Worker-4:work-29) LocalStoragePoolAllocator trying to find storage pool
> to fit the vm
> 2013-11-25 15:41:12,103 DEBUG
> [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29)
> ClusterScopeStoragePoolAllocator looking for storage pool
> 2013-11-25 15:41:12,103 DEBUG
> [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29)
> Looking for pools in dc: 1 pod:1 cluster:1
> 2013-11-25 15:41:12,107 DEBUG
> [storage.allocator.AbstractStoragePoolAllocator] (HA-Worker-4:work-29)
> Checking if storage pool is suitable, name: null ,poolId: 200
> 2013-11-25 15:41:12,111 DEBUG [cloud.storage.StorageManagerImpl]
> (HA-Worker-4:work-29) Checking pool 200 for storage, totalSize:
> 11810778316800, usedBytes: 9755417411584, usedPct: 0.8259758290194649,
> disable threshold: 0.1
> 2013-11-25 15:41:12,111 DEBUG [cloud.storage.StorageManagerImpl]
> (HA-Worker-4:work-29) Insufficient space on pool: 200 since its usage
> percentage: 0.8259758290194649 has crossed the
> pool.storage.capacity.disablethreshold: 0.1
> 2013-11-25 15:41:12,111 DEBUG
> [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29)
> ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
> 2013-11-25 15:41:12,111 DEBUG
> [storage.allocator.ZoneWideStoragePoolAllocator] (HA-Worker-4:work-29)
> ZoneWideStoragePoolAllocator to find storage pool
> 2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) No suitable pools found for volume: Vol[13|vm=12|ROOT]
> under cluster: 1
> 2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) No suitable pools found
> 2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) No suitable storagePools found under this Cluster: 1
> 2013-11-25 15:41:12,117 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-29) Could not find suitable Deployment Destination for this
> VM under any clusters, returning.
> -2013-11-25 15:32:09,784 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (HA-Worker-3:work-27) Unable to find destination for migrating the vm
> VM[User|z1V5]
> 2013-11-25 15:32:09,784 WARN [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-3:work-27) Insufficient capacity for migrating a VM.
> 2013-11-25 15:32:09,784 DEBUG [cloud.deploy.FirstFitPlanner]
> (HA-Worker-1:work-29) The specified cluster is in avoid set, returning.
> 2013-11-25 15:32:09,784 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (HA-Worker-1:work-29) Unable to find destination for migrating the vm
> VM[SecondaryStorageVm|s-12-VM]
> 2013-11-25 15:32:09,784 WARN [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-1:work-29) Insufficient capacity for migrating a VM.
> 2013-11-25 15:32:09,786 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
> (HA-Worker-4:work-28) Could not find suitable Deployment Destination for this
> VM under any clusters, returning.
> 2013-11-25 15:32:09,786 DEBUG [cloud.deploy.FirstFitPlanner]
> (HA-Worker-4:work-28) Searching resources only under specified Cluster: 1
> 2013-11-25 15:32:09,788 DEBUG [cloud.resource.ResourceManagerImpl]
> (HA-Worker-3:work-27) No next resource state for host 5 while current state
> is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for
> current state =ErrorInMaintenance event =UnableToMigrate
> at
> com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at
> com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at
> com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
> at
> com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-11-25 15:32:09,789 INFO [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-3:work-27) Rescheduling HAWork[27-Migration-8-Running-Migrating]
> to try again at Mon Nov 25 15:34:11 PST 2013
> 2013-11-25 15:32:09,790 DEBUG [cloud.deploy.FirstFitPlanner]
> (HA-Worker-4:work-28) The specified cluster is in avoid set, returning.
> 2013-11-25 15:32:09,790 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (HA-Worker-4:work-28) Unable to find destination for migrating the vm
> VM[User|z1V6]
> 2013-11-25 15:32:09,790 WARN [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-4:work-28) Insufficient capacity for migrating a VM.
> 2013-11-25 15:32:09,790 DEBUG [cloud.resource.ResourceManagerImpl]
> (HA-Worker-1:work-29) No next resource state for host 5 while current state
> is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for
> current state =ErrorInMaintenance event =UnableToMigrate
> at
> com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at
> com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at
> com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
> at
> com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-11-25 15:32:09,790 INFO [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-1:work-29) Rescheduling HAWork[29-Migration-12-Running-Migrating]
> to try again at Mon Nov 25 15:34:11 PST 2013
> 2013-11-25 15:32:09,793 DEBUG [cloud.resource.ResourceManagerImpl]
> (HA-Worker-4:work-28) No next resource state for host 5 while current state
> is ErrorInMaintenance with event UnableToMigrate
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for
> current state =ErrorInMaintenance event =UnableToMigrate
> at
> com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
> at
> com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
> at
> com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
> at
> com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
> 2013-11-25 15:32:09,793 INFO [cloud.ha.HighAvailabilityManagerImpl]
> (HA-Worker-4:work-28) Rescheduling HAWork[28-Migration-9-Running-Migrating]
> to try again at Mon Nov 25 15:34:11 PST 2013
> 2013-11-25 15:32:15,047 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-2:null) HostStatsCollector is running...
> 2013-11-25 15:32:15,059 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-85:null) Seq 1-2115240044: Executing request
> 2013-11-25 15:32:15,162 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-6:null) SeqA 3-1080: Processing Seq 3-1080: { Cmd ,
> MgmtId: -1, via: 3, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand
> ":{"_proxyVmId":2,"_loadInfo":"
> {\n \"connections\": []\n}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
