Without looking too deep into this it seems that there maybe a stuck task
trying to snapshot the VHD.
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
Again, based on what you sent, I would look at any hung xapi tasks on the host
in question.
Regards,
Marty Godsey
nSource Solutions
-----Original Message-----
From: J Andersons
Sent: Friday, April 21, 2017 2:20 AM
To: users@cloudstack.apache.org
Subject: Unable to start a VM due to insufficient capacity
Hi!
Just like that it is impossible to start any vm. ACS4.8, Xenservers 6.5 Unable
to start a VM due to insufficient capacity. I can see in xencenter it is trying
to start VM on all hosts but fails. Also there is enough resources on hosts.
Maybe its an XenServer issue, because it is also unable to stop VMS.
In log when starting VM:
2017-04-21 08:50:35,806 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Task failed! Task
record: uuid: a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
2017-04-21 08:50:35,818 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:461)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4807)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,826 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) unable to destroy
task(com.xensource.xenapi.Task@27d940e) on
host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to You gave an invalid object
reference. The object may have recently been deleted. The class parameter
gives the type of reference given, and the handle parameter echoes the bad
value given.
2017-04-21 08:50:35,826 WARN [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Catch Exception: class
com.cloud.utils.exception.CloudRuntimeException due to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,828 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to start i-95-292-VM due
to
com.cloud.utils.exception.CloudRuntimeException: Unable to start
VM(i-95-292-VM) on host(a6601054-b730-460a-b654-a29f1a5b8a1c) due to
Task failed! Task record: uuid:
a9ac4c12-a6ba-6d9a-556f-b5d88f8cd2d3
nameLabel: Async.VM.start_on
nameDescription:
allowedOperations: []
currentOperations: {}
created: Fri Apr 21 08:48:34 EEST 2017
finished: Fri Apr 21 08:50:35 EEST 2017
status: failure
residentOn: com.xensource.xenapi.Host@c07df7eb
progress: 1.0
type: <none/>
result:
errorInfo: [OTHER_OPERATION_IN_PROGRESS, VDI,
OpaqueRef:1d681250-d950-81ec-9f87-f7eb3d8b0cc1, snapshot]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:4826)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:126)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,889 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,901 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to clean up VBD due to
You gave an invalid object reference. The object may have recently been
deleted. The class parameter gives the type of reference given, and the handle
parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VBD.unplug(VBD.java:1109)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3445)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:35,913 WARN [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) Unable to cleanup VIF You gave
an invalid object reference. The object may have recently been deleted. The
class parameter gives the type of reference given, and the handle parameter
echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:693)
at com.xensource.xenapi.Connection.dispatch(Connection.java:395)
at
com.cloud.hypervisor.xenserver.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:457)
at com.xensource.xenapi.VIF.unplug(VIF.java:921)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:3453)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:208)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixStartCommandWrapper.execute(CitrixStartCommandWrapper.java:53)
at
com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
at
com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
at
com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2017-04-21 08:50:36,091 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-28:ctx-6bbd1566) (logid:91c36627) The VM is in stopped state,
detected problem during startup : i-95-292-VM
--
J Andersons
