On 01/15/2013 04:47 PM, Anthony Xu wrote:
destroy_domain is a very low level command, it talks to Xen hypervisor
directly. The command hangs might mean it is waiting for a lock in hypervisor
which is held by other command/xapi.
Try restaringing xapi and destroy_domain.
xe-toolstack-restart
if it doesn't work, the only solution I know is to reboot the host, you can
migrate VMs to other hosts before you reboot this host.
Anthony
I restarted the toolstack, then tried to migrate a VM poff of it, and
here is what I got in the management log:
2013-01-15 18:05:12,394 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-12:job-715) CPU STATS after allocation: for host: 6, old
used: 0, old reserved: 0, actual total: 57456, total with
overprovisioning: 71820; new used:500, reserved:0; requested
cpu:500,alloc_from_last:false
2013-01-15 18:05:12,394 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-12:job-715) RAM STATS after allocation: for host: 6, old
used: 0, old reserved: 0, total: 98962234560; new used: 536870912,
reserved: 0; requested mem: 536870912,alloc_from_last:false
2013-01-15 18:05:12,399 DEBUG [agent.transport.Request]
(Job-Executor-12:job-715) Seq 1-1375404813: Sending { Cmd , MgmtId:
130577622632, via: 1, Ver: v1, Flags: 100111,
[{"MigrateCommand":{"vmName":"i-17-67-VM","destIp":"172.16.5.5","hostGuid":"e1deec38-20f5-46e1-aa11-7d18ce7bd97c","isWindows":false,"wait":0}}]
}
2013-01-15 18:05:12,399 DEBUG [agent.transport.Request]
(Job-Executor-12:job-715) Seq 1-1375404813: Executing: { Cmd , MgmtId:
130577622632, via: 1, Ver: v1, Flags: 100111,
[{"MigrateCommand":{"vmName":"i-17-67-VM","destIp":"172.16.5.5","hostGuid":"e1deec38-20f5-46e1-aa11-7d18ce7bd97c","isWindows":false,"wait":0}}]
}
2013-01-15 18:05:12,399 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-483:null) Seq 1-1375404813: Executing request
2013-01-15 18:05:12,567 DEBUG [xen.resource.CitrixResourceBase]
(DirectAgent-483:null) 5. The VM i-17-67-VM is in Stopping state
2013-01-15 18:05:14,020 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-111:null) Ping from 6
2013-01-15 18:05:14,361 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-145:null) Ping from 3
2013-01-15 18:05:15,923 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:19,650 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:23,217 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-337:null) Ping from 1
2013-01-15 18:05:35,161 DEBUG
[storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone
1 is ready to launch secondary storage VM
2013-01-15 18:05:35,268 DEBUG
[cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone
1 is ready to launch console proxy
2013-01-15 18:05:35,483 DEBUG
[network.router.VirtualNetworkApplianceManagerImpl]
(RouterStatusMonitor-1:null) Found 8 routers.
2013-01-15 18:05:37,448 WARN [agent.manager.DirectAgentAttache]
(DirectAgent-97:null) Seq 1-1375404807: Exception Caught while executing
command
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5after 30 retry
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.PoolEmergencyResetMaster(XenServerConnectionPool.java:439)
at
com.cloud.hypervisor.xen.resource.XenServerConnectionPool.connect(XenServerConnectionPool.java:651)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.getConnection(CitrixResourceBase.java:4872)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:1749)
at
com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:404)
at
com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:69)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:187)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-01-15 18:05:37,449 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-97:null) Seq 1-1375404807: Response Received:
2013-01-15 18:05:37,449 DEBUG [agent.transport.Request]
(DirectAgent-97:null) Seq 1-1375404807: Processing: { Ans: , MgmtId:
130577622632, via: 1, Ver: v1, Flags: 10,
[{"Answer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException:
Unable to reset master of slave 172.16.5.3 to 172.16.5.5 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5after 30 retry","wait":0}}] }
2013-01-15 18:05:37,449 WARN [cloud.vm.VirtualMachineManagerImpl]
(DirectAgent-97:null) Cleanup failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5after 30 retry
2013-01-15 18:05:37,449 DEBUG [agent.transport.Request]
(StatsCollector-3:null) Seq 1-1375404807: Received: { Ans: , MgmtId:
130577622632, via: 1, Ver: v1, Flags: 10, { Answer } }
2013-01-15 18:05:37,449 WARN [cloud.vm.VirtualMachineManagerImpl]
(StatsCollector-3:null) Cleanup failed due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5after 30 retry
2013-01-15 18:05:37,449 DEBUG [agent.manager.AgentManagerImpl]
(StatsCollector-3:null) Details from executing class
com.cloud.agent.api.GetHostStatsCommand:
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5 due to
com.cloud.utils.exception.CloudRuntimeException: Unable to reset master
of slave 172.16.5.3 to 172.16.5.5after 30 retry
-----Original Message-----
From: Nik Martin [mailto:nik.mar...@nfinausa.com]
Sent: Tuesday, January 15, 2013 2:40 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Console Proxy VM unavailable
On 01/15/2013 04:28 PM, Anthony Xu wrote:
Hi Nik,
List_domains and destroy_domain are working on host not pool.
So if you have multiple hosts in a XenServer pool , you might need to
execute above command on that host,
You can try following to find the host
Xe vm-param-list uuid="vm uuid" | grep resident-on
You will get the host uuid
Xe host-param-list uuid="above host uuid"
Then you get the host information.
Log into the host and retry the procedure to destroy VM.
I ran that command on the VM in question, and it hangs, on any host in
the pool. The command works fine for any other VM. This VM must be in
bad shape.
Anthony
-----Original Message-----
From: Nik Martin [mailto:nik.mar...@nfinausa.com]
Sent: Tuesday, January 15, 2013 2:17 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Console Proxy VM unavailable
On 01/15/2013 03:55 PM, Anthony Xu wrote:
Hi Nik,
Try this
http://support.citrix.com/article/CTX131421
Anthony
Anthony,
Running list_domains with the VMs UUID does not return any results,
but
if I just run list_domains, I get all the control domains, and one
odd
one:
117 | deadbeef-dead-beef-dead-beef00000075 | DS
could this be the control domain for the hung VM?
-----Original Message-----
From: Nik Martin [mailto:nik.mar...@nfinausa.com]
Sent: Tuesday, January 15, 2013 1:41 PM
To: cloudstack-users@incubator.apache.org
Subject: Re: Console Proxy VM unavailable
A bit more information, I logged in to Xen center and the VM is
hung
with a yellow arrow. There are no options to shutdown, reboot, or
anything. How do I force this VM off?
On 01/15/2013 03:32 PM, Nik Martin wrote:
I have a CS 3.02 cloud in production, and it has been running
fine
since
December 1. Today, I tried to use the web console, and it times
out.
I
turned debug up on the management server, and see the command to
launch
the proxy go by, but no errors. I tried to migrate it, and now
it's
hung in a migrating state. What is the best way to reset this VM
and
get back on track, and also diagnose why this happened?
--
Regards,
Nik
Nik Martin
nfina Technologies, Inc.
+1.251.243.0043 x1003
http://nfinausa.com
Relentless Reliability
--
Regards,
Nik
Nik Martin
nfina Technologies, Inc.
+1.251.243.0043 x1003
http://nfinausa.com
Relentless Reliability
--
Regards,
Nik
Nik Martin
nfina Technologies, Inc.
+1.251.243.0043 x1003
http://nfinausa.com
Relentless Reliability
--
Regards,
Nik
Nik Martin
nfina Technologies, Inc.
+1.251.243.0043 x1003
http://nfinausa.com
Relentless Reliability
