When you say there is enough resources on hosts, exactly what’s memory and cpu
allocation on each of the hosts? Unless you have updated following global
settings, the default settings are to disable VM allocation if cluster’s
overall CPU or memory allocation is over 75%, and you will get insufficient
resources error when you creating new instances.
cluster.cpu.allocated.capacity.disablethreshold = 0.75
cluster.memory.allocated.capacity.disablethreshold = 0.75
There are other situations where you could also receive this error, such as if
you specified wrong network or the tagged resources are not available (storage,
host, etc, or wrong tag was specified) and so on. This error message is pretty
much useless as it is given, not to cloud operators or administrators anyway.
Yiping
On 4/21/17, 4:21 PM, "Marty Godsey" <ma...@gonsource.com> wrote:
Without looking too deep into this it seems that there maybe a stuck task
trying to snapshot the VHD.
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
Again, based on what you sent, I would look at any hung xapi tasks on the
host in question.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: J Andersons
Sent: Friday, April 21, 2017 2:20 AM
To: users@cloudstack.apache.org
Subject: Unable to start a VM due to insufficient capacity
Hi!
Just like that it is impossible to start any vm. ACS4.8, Xenservers 6.5
Unable to start a VM due to insufficient capacity. I can see in xencenter it is
trying to start VM on all hosts but fails. Also there is enough resources on
hosts.
Maybe its an XenServer issue, because it is also unable to stop VMS.
In log when starting VM:
2017-04-21 08:50:35,806 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Task failed! Task
record: uuid: a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
2017-04-21 08:50:35,818 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:461)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4807)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,826 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) unable to destroy
task(com.xensource.xenapi.Task@27d940e) on
host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to You gave an invalid
object reference. The object may have recently been deleted. The class
parameter gives the type of reference given, and the handle parameter echoes
the bad value given.
2017-04-21 08:50:35,826 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Catch Exception: class
com.cloud.utils.exception.CloudRuntimeException due to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,828 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start i-95-292-VM
due to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,889 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due
to You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,901 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due
to You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,913 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to cleanup VIF You
gave an invalid object reference. The object may have recently been deleted.
The class parameter gives the type of reference given, and the handle parameter
echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VIF.unplug(VIF.java:921)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3453)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:36,091 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) The VM is in stopped state,
detected problem during startup : i-95-292-VM
--
J Andersons
