Ameen Ali created CLOUDSTACK-6423:
-------------------------------------
Summary: No clusters found having a host with enough capacity
Key: CLOUDSTACK-6423
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6423
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0, 4.2.1
Environment: Centos 6.5 Master
Reporter: Ameen Ali
Fix For: 4.2.0
I have just installed CloudStack 4.2.1 as management server on my Centos 6.5. I
am having issues with starting the secondary storage VMs. Here are the messages
from the log /var/log/cloudstack/management/management-server.log :
2014-04-16 03:22:00,789 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 routers to update status.
2014-04-16 03:22:00,790 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2014-04-16 03:22:06,032 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Resetting hosts suitable for reconnect
2014-04-16 03:22:06,035 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed resetting hosts suitable for reconnect
2014-04-16 03:22:06,035 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Acquiring hosts for clusters already owned by this management server
2014-04-16 03:22:06,038 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed acquiring hosts for clusters already owned by this
management server
2014-04-16 03:22:06,038 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Acquiring hosts for clusters not owned by any management server
2014-04-16 03:22:06,039 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager
Timer:null) Completed acquiring hosts for clusters not owned by any management
server
2014-04-16 03:22:08,039 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-27:null) Ping from 1
2014-04-16 03:22:09,155 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-16:null) Seq 1-2066939909: Executing request
2014-04-16 03:22:09,434 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-16:null) Seq 1-2066939909: Response Received:
2014-04-16 03:22:09,435 DEBUG [agent.transport.Request] (DirectAgent-16:null)
Seq 1-2066939909: Processing: { Ans: , MgmtId: 52240630156, via: 1, Ver: v1,
Flags: 10,
[{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}]
}
2014-04-16 03:22:28,685 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-04-16 03:22:28,687 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null) No
running secondary storage vms found in datacenter id=1, starting one
2014-04-16 03:22:28,690 INFO [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) No stopped secondary storage vm is available, need to
allocate a new secondary storage vm
2014-04-16 03:22:28,691 DEBUG [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Assign secondary storage vm from a newly started instance
for request from data center : 1
2014-04-16 03:22:28,701 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network
Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]: Ntwk[204|Guest|6]
2014-04-16 03:22:28,702 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,704 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network
Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2014-04-16 03:22:28,704 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,706 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network
Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2014-04-16 03:22:28,706 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,708 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Found existing network configuration for offering [Network
Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2014-04-16 03:22:28,708 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Releasing lock for
Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,716 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating entries for VM: VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,719 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,719 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in
network Ntwk[204|Guest|6] with requested profile NicProfile[0-0-null-null-null
2014-04-16 03:22:28,725 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in
network Ntwk[202|Control|3] with requested profile null
2014-04-16 03:22:28,729 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in
network Ntwk[201|Management|2] with requested profile null
2014-04-16 03:22:28,734 DEBUG [cloud.network.NetworkManagerImpl]
(secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in
network Ntwk[203|Storage|4] with requested profile null
2014-04-16 03:22:28,739 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,765 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Allocation completed for VM: VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,766 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) received secondary storage vm alert
2014-04-16 03:22:28,767 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter]
(secstorage-1:null) New secondary storage vm created, zone: Zone1,
secStorageVm: s-316-VM, public IP: null, private IP: null
2014-04-16 03:22:28,771 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Stopped to Starting with event:
StartRequestedvm's original host id: null new host id: null host id before
state transition: null
2014-04-16 03:22:28,771 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Successfully transitioned to start state for
VM[SecondaryStorageVm|s-316-VM] reservation id =
ddfc829e-a6fc-4e39-9a5b-13244f1dd013
2014-04-16 03:22:28,774 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2014-04-16 03:22:28,774 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,775 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_9c406fc8@6337c832
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Searching all possible resources under this Zone: 1
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) Listing clusters in order of aggregate capacity, that have
(atleast one host with) enough CPU and RAM capacity under this Zone: 1
2014-04-16 03:22:28,779 DEBUG [cloud.deploy.FirstFitPlanner]
(secstorage-1:null) No clusters found having a host with enough capacity,
returning.
2014-04-16 03:22:28,784 DEBUG [cloud.capacity.CapacityManagerImpl]
(secstorage-1:null) VM state transitted from :Starting to Stopped with event:
OperationFailedvm's original host id: null new host id: null host id before
state transition: null
2014-04-16 03:22:28,786 WARN [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[SecondaryStorageVm|s-316-VM]Scope=interface
com.cloud.dc.DataCenter; id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-16 03:22:28,787 INFO [storage.secondary.SecondaryStorageManagerImpl]
(secstorage-1:null) Unable to start secondary storage vm for standby capacity,
secStorageVm vm Id : 316, will recycle it and start a new one
2014-04-16 03:22:28,787 INFO
[cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null)
Primary secondary storage is not even started, wait until next turn
2014-04-16 03:22:28,923 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-04-16 03:22:28,923 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Expand console proxy standby capacity for zone Zone1
2014-04-16 03:22:28,925 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Found a stopped console proxy, bring it up to running
pool. proxy vm id : 2
2014-04-16 03:22:28,929 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Stopped to Starting with event:
StartRequestedvm's original host id: null new host id: null host id before
state transition: null
2014-04-16 03:22:28,929 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Successfully transitioned to start state for
VM[ConsoleProxy|v-2-VM] reservation id = b95325ab-1513-4131-9611-a3be07eb0b4a
2014-04-16 03:22:28,932 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2014-04-16 03:22:28,932 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,933 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) DeploymentPlanner allocation algorithm:
com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_9c406fc8@6337c832
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 500, requested ram: 1073741824
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl]
(consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Searching all possible resources under this Zone: 1
2014-04-16 03:22:28,935 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) Listing clusters in order of aggregate capacity, that
have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2014-04-16 03:22:28,936 DEBUG [cloud.deploy.FirstFitPlanner]
(consoleproxy-1:null) No clusters found having a host with enough capacity,
returning.
2014-04-16 03:22:28,946 DEBUG [cloud.capacity.CapacityManagerImpl]
(consoleproxy-1:null) VM state transitted from :Starting to Stopped with event:
OperationFailedvm's original host id: null new host id: null host id before
state transition: null
2014-04-16 03:22:28,947 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl]
(consoleproxy-1:null) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a
deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter;
id=1
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
at
com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
I tried checking the op_host_capacity table from the db, here it is:
+----+---------+----------------+--------+------------+---------------+-------------------+----------------+---------------+----------------+---------------------+---------------------+
| id | host_id | data_center_id | pod_id | cluster_id | used_capacity |
reserved_capacity | total_capacity | capacity_type | capacity_state |
update_time | created |
+----+---------+----------------+--------+------------+---------------+-------------------+----------------+---------------+----------------+---------------------+---------------------+
| 1 | NULL | 1 | NULL | NULL | 0 |
0 | 50 | 8 | Enabled | 2014-03-26
20:15:28 | 2014-03-26 20:10:28 |
| 2 | NULL | 1 | 1 | NULL | 1 |
0 | 253 | 5 | Enabled | 2014-03-26
20:20:28 | 2014-03-26 20:10:28 |
| 3 | 1 | 1 | 1 | 1 | 0 |
0 | 148180032 | 0 | Enabled | 2014-04-16
00:24:01 | 2014-03-26 20:18:01 |
| 4 | 1 | 1 | 1 | 1 | 0 |
0 | 2892 | 1 | Enabled | 2014-03-26
20:18:01 | 2014-03-26 20:18:01 |
| 5 | 2 | 1 | 1 | 1 | 0 |
0 | 37004247040 | 3 | Enabled | 2014-03-26
20:19:11 | 2014-03-26 20:19:11 |
+----+---------+----------------+--------+------------+---------------+-------------------+----------------+---------------+----------------+---------------------+---------------------+
5 rows in set (0.00 sec)
I have not used any of my capacity yet I still cannot start my system VM.
Please help. Thanks
--
This message was sent by Atlassian JIRA
(v6.2#6252)
