[
https://issues.apache.org/jira/browse/CLOUDSTACK-1508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13593331#comment-13593331
]
Devdeep Singh commented on CLOUDSTACK-1508:
-------------------------------------------
Took a look at the logs and the host on which the problem was reproduced
(10.147.40.7 and 10.147.40.18). It looks like the vhd-util tool wasn't copied
to the host. It isn't present in the /opt/xensource/bin directory on the host.
This is the vhd-util tool the copy_vhd_from_secondarystorage script refers to.
When cloudstack adds a host it'll copy the vhd-util tool to the directory and
tag the host. Subsequently it won't try to copy the script as the host is
tagged and cloudstack management server assumes the host to be already setup.
When cloudstack is setup from the package instructions are given to download
and copy the vhd-util tool to a certain folder. When a host is added, all the
tool/scripts from this directory are copied to the host and the host is tagged.
If the vhd-util tool was downloaded and copied, the above issue can happen.
In the logs it can be seen that copy_vhd_from_secondarystorage scipt reported
error just before VM.start error. So the issue looks like is that vhd-util
wasn't downloaded and management server never copied it the host.
To recover from this, you can remove the tags from host. Download and copy the
vhd-util tool as per the instructions in cloudstack setup and restart the
management server.
> systems vms are fail to start
> -----------------------------
>
> Key: CLOUDSTACK-1508
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-1508
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: sadhu suresh
> Assignee: Devdeep Singh
> Priority: Blocker
> Attachments: management-server.rar
>
>
> Hypervisor:xen6.0.2
> Once enabled the zone,Noticed below exception in the management log when
> system VMs are in starting state and went to stopped state due to below
> exceptions.
> system vms are continuously failing with below exceptions.
> Task failed! Task record: uuid:
> d95cdf25-c7bb-ffe1-33f3-8f9324eddff2
> nameLabel: Async.VM.start_on
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Fri Mar 01 11:05:09 EST 2013
> finished: Fri Mar 01 11:05:11 EST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@31c6e197
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [Traceback (most recent call last):, File
> "/usr/bin/pygrub", line 808, in ?, fs = fsimage.open(file, part_offs[0],
> bootfsoptions), IOError: [Errno 95] Operation not supported, ]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3283)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3395)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1424)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,636 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Catch Exception: class
> com.cloud.utils.exception.CloudRuntimeException due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(s-14-VM)
> on host(511ec001-1a19-400e-8e7e-11516d685475) due to Task failed! Task
> record: uuid: d95cdf25-c7bb-ffe1-33f3-8f9324eddff2
> nameLabel: Async.VM.start_on
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Fri Mar 01 11:05:09 EST 2013
> finished: Fri Mar 01 11:05:11 EST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@31c6e197
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [Traceback (most recent call last):, File
> "/usr/bin/pygrub", line 808, in ?, fs = fsimage.open(file, part_offs[0],
> bootfsoptions), IOError: [Errno 95] Operation not supported, ]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(s-14-VM)
> on host(511ec001-1a19-400e-8e7e-11516d685475) due to Task failed! Task
> record: uuid: d95cdf25-c7bb-ffe1-33f3-8f9324eddff2
> nameLabel: Async.VM.start_on
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Fri Mar 01 11:05:09 EST 2013
> finished: Fri Mar 01 11:05:11 EST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@31c6e197
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [Traceback (most recent call last):, File
> "/usr/bin/pygrub", line 808, in ?, fs = fsimage.open(file, part_offs[0],
> bootfsoptions), IOError: [Errno 95] Operation not supported, ]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3406)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1424)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,637 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to start s-14-VM due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to start VM(s-14-VM)
> on host(511ec001-1a19-400e-8e7e-11516d685475) due to Task failed! Task
> record: uuid: d95cdf25-c7bb-ffe1-33f3-8f9324eddff2
> nameLabel: Async.VM.start_on
> nameDescription:
> allowedOperations: []
> currentOperations: {}
> created: Fri Mar 01 11:05:09 EST 2013
> finished: Fri Mar 01 11:05:11 EST 2013
> status: failure
> residentOn: com.xensource.xenapi.Host@31c6e197
> progress: 1.0
> type: <none/>
> result:
> errorInfo: [Traceback (most recent call last):, File
> "/usr/bin/pygrub", line 808, in ?, fs = fsimage.open(file, part_offs[0],
> bootfsoptions), IOError: [Errno 95] Operation not supported, ]
> otherConfig: {}
> subtaskOf: com.xensource.xenapi.Task@aaf13f6f
> subtasks: []
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.startVM(CitrixResourceBase.java:3406)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1424)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,684 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 routers to update status.
> 2013-03-01 10:58:47,688 DEBUG
> [network.router.VirtualNetworkApplianceManagerImpl]
> (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
> 2013-03-01 10:58:47,743 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to clean up VBD due to You gave an invalid
> object reference. The object may have recently been deleted. The class
> parameter gives the type of reference given, and the handle parameter echoes
> the bad value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VBD.unplug(VBD.java:1059)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1224)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,756 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to clean up VBD due to You gave an invalid
> object reference. The object may have recently been deleted. The class
> parameter gives the type of reference given, and the handle parameter echoes
> the bad value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VBD.unplug(VBD.java:1059)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1224)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,768 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to cleanup VIF You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VIF.unplug(VIF.java:878)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1232)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,790 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to cleanup VIF You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VIF.unplug(VIF.java:878)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1232)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,802 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to cleanup VIF You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VIF.unplug(VIF.java:878)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1232)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:47,814 WARN [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) Unable to cleanup VIF You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> at com.xensource.xenapi.Types.checkResponse(Types.java:209)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
> at
> com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
> at com.xensource.xenapi.VIF.unplug(VIF.java:878)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.handleVmStartFailure(CitrixResourceBase.java:1232)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1487)
> at
> com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:529)
> at
> com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:73)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-03-01 10:58:48,342 INFO [xen.resource.XenServer56Resource]
> (DirectAgent-18:null) Catch com.xensource.xenapi.Types$InvalidValue: failed
> to destory VLAN eth0 on host 511ec001-1a19-400e-8e7e-11516d685475 due to The
> value given is invalid
> 2013-03-01 10:58:48,344 DEBUG [xen.resource.CitrixResourceBase]
> (DirectAgent-18:null) The VM is in stopped state, detected problem during
> startup : s-14-VM
> 2013-03-01 10:58:48,344 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-18:null) Seq 1-1527644195: Cancelling because one of the answers
> is false
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
