JF Vincent created CLOUDSTACK-7703:
--------------------------------------
Summary: Cloudstack server endless loop when trying to create a
volume while storage pool is full
Key: CLOUDSTACK-7703
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-7703
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.3.0
Environment: Centos 6.5
Reporter: JF Vincent
Priority: Critical
When trying to create a VM, and thus a volume for it and the primary storage is
full (over 90%), the managament server enter in and endless loop (extract
below) and we have to restart it to exit this loop.
2014-10-14 11:39:20,701 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable pools found for volume:
Vol[5436|vm=5855|DATADISK] under cluster: 2
2014-10-14 11:39:20,702 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable pools found
2014-10-14 11:39:20,702 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) No suitable storagePools found
under this Cluster: 2
2014-10-14 11:39:20,705 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Could not find suitable Deployment
Destination for this VM under any clusters, returning.
2014-10-14 11:39:20,705 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Searching all possible resources
under this Zone: 2
2014-10-14 11:39:20,705 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Listing clusters in order of
aggregate capacity, that have (atleast one host with) enough CPU and RAM
capacity under this Zone: 2
2014-10-14 11:39:20,707 DEBUG [cloud.deploy.FirstFitPlanner]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Removing from the clusterId list
these clusters from avoid set: []
2014-10-14 11:39:20,714 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c) Checking resources in Cluster: 2
under Pod: 2
2014-10-14 11:39:20,714 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Looking
for hosts in dc: 2 pod:2 cluster:2
2014-10-14 11:39:20,716 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator)
FirstFitAllocator has 3 hosts to check for allocation: [Host[-79-Routing],
Host[-89-Routing], Host[-77-Routing]]
2014-10-14 11:39:20,717 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found 3
hosts for allocation after prioritization: [Host[-79-Routing],
Host[-89-Routing], Host[-77-Routing]]
2014-10-14 11:39:20,717 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Looking
for speed=500Mhz, Ram=500
2014-10-14 11:39:20,720 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 79
has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and
requested speed: 500
2014-10-14 11:39:20,720 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking
if host: 79 has enough capacity for requested CPU: 500 and requested RAM:
524288000 , cpuOverprovisioningFactor: 4.0
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's
actual total CPU: 19192 and CPU after applying overprovisioning: 76768
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU:
57268 , Requested CPU: 500
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM:
93916725248 , Requested RAM: 524288000
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has
enough CPU and RAM available
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc CPU from host: 79, used: 19000, reserved: 500, actual total: 19192, total
with overprovisioning: 76768; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2014-10-14 11:39:20,721 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc MEM from host: 79, used: 17192452096, reserved: 524288000, total:
111633465344; requested mem: 524288000,alloc_from_last_host?:false
,considerReservedCapacity?: true
2014-10-14 11:39:20,721 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a
suitable host, adding to list: 79
2014-10-14 11:39:20,724 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 89
has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and
requested speed: 500
2014-10-14 11:39:20,724 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking
if host: 89 has enough capacity for requested CPU: 500 and requested RAM:
524288000 , cpuOverprovisioningFactor: 4.0
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's
actual total CPU: 19192 and CPU after applying overprovisioning: 76768
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU:
66268 , Requested CPU: 500
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM:
101013487616 , Requested RAM: 524288000
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has
enough CPU and RAM available
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc CPU from host: 89, used: 10500, reserved: 0, actual total: 19192, total
with overprovisioning: 76768; requested cpu:500,alloc_from_last_host?:false
,considerReservedCapacity?: true
2014-10-14 11:39:20,725 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc MEM from host: 89, used: 10619977728, reserved: 0, total: 111633465344;
requested mem: 524288000,alloc_from_last_host?:false
,considerReservedCapacity?: true
2014-10-14 11:39:20,725 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a
suitable host, adding to list: 89
2014-10-14 11:39:20,727 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host: 77
has cpu capability (cpu:8, speed:2399) to support requested CPU: 1 and
requested speed: 500
2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Checking
if host: 77 has enough capacity for requested CPU: 500 and requested RAM:
524288000 , cpuOverprovisioningFactor: 4.0
2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Hosts's
actual total CPU: 19192 and CPU after applying overprovisioning: 76768
2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free CPU:
59768 , Requested CPU: 500
2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Free RAM:
95586058240 , Requested RAM: 524288000
2014-10-14 11:39:20,728 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Host has
enough CPU and RAM available
2014-10-14 11:39:20,729 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc CPU from host: 77, used: 11500, reserved: 5500, actual total: 19192,
total with overprovisioning: 76768; requested
cpu:500,alloc_from_last_host?:false ,considerReservedCapacity?: true
2014-10-14 11:39:20,729 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) STATS: Can
alloc MEM from host: 77, used: 10280239104, reserved: 5767168000, total:
111633465344; requested mem: 524288000,alloc_from_last_host?:false
,considerReservedCapacity?: true
2014-10-14 11:39:20,729 DEBUG [allocator.impl.FirstFitAllocator]
(Job-Executor-10:ctx-02d42f8f ctx-e581af2c FirstFitRoutingAllocator) Found a
suitable host, adding to list: 77
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
