[
https://issues.apache.org/jira/browse/CLOUDSTACK-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13704441#comment-13704441
]
Koushik Das commented on CLOUDSTACK-3248:
-----------------------------------------
The correct way to remove a host is to first put it in maintenance and then
remove it. This ensures that all VMs running in it are either migrated or
stopped. This check is currently only enforced in UI. Using the API directly a
host can be deleted without putting it in maintenance. This leaves the system
in a bad state wherein the host is no longer there but VMs running on it are
still present. Not sure as to why this wasn't found earlier.
> XenServer Host got removed successfully inspite of running VMs on the host
> --------------------------------------------------------------------------
>
> Key: CLOUDSTACK-3248
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-3248
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Chandan Purushothama
> Assignee: Koushik Das
> Priority: Critical
> Fix For: 4.2.0
>
>
> ==============
> From the API log:
> ==============
> 2013-06-27 11:48:36,855 INFO [cloud.api.ApiServer] (catalina-exec-15:null)
> (userId=2 accountId=2 sessionId=62A69AC91786781EA9BC2F5E534F5590)
> 10.252.120.177 -- GET
> command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> 200 { "deletehostresponse" : { "success" : "true"} }
> ==========================
> From the Management Server Log:
> ==========================
> 2013-06-27 11:48:36,765 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
> ===START=== 10.252.120.177 -- GET
> command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> ConsoleProxyManagerImpl_EnhancerByCloudStack_edb8e808
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_9f8bf2f0
> 2013-06-27 11:48:36,800 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> JuniperSRXExternalFirewallElement_EnhancerByCloudStack_c0d55fca
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> KvmServerDiscoverer_EnhancerByCloudStack_6a93cc3b
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> VmwareServerDiscoverer_EnhancerByCloudStack_c0f46d01
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> NetscalerElement_EnhancerByCloudStack_2c0cf22
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> NetworkUsageManagerImpl_EnhancerByCloudStack_fed7434f
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> F5ExternalLoadBalancerElement_EnhancerByCloudStack_91202979
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> NiciraNvp
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> CiscoVnmcElement_EnhancerByCloudStack_439d7cf0
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,801 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> OvmDiscoverer_EnhancerByCloudStack_d6ea6366
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> LxcServerDiscoverer_EnhancerByCloudStack_2f0a46f8
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Dispatching resource state event DELETE_HOST to
> XcpServerDiscoverer_EnhancerByCloudStack_6d80ebbf
> 2013-06-27 11:48:36,802 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Deleting Host: 5
> Guid:b213f6c1-16d6-4fa3-8998-dce12a65d1e3
> 2013-06-27 11:48:36,806 DEBUG [dc.dao.DataCenterIpAddressDaoImpl]
> (catalina-exec-15:null) Releasing ip address: 10.223.57.194 data center 1
> 2013-06-27 11:48:36,810 DEBUG [cloud.host.Status] (catalina-exec-15:null)
> Transition:[Resource state = Enabled, Agent event = Remove, Host id = 5, name
> = Rack3Host8.lab.vmops.com]
> 2013-06-27 11:48:36,812 DEBUG [cloud.host.Status] (catalina-exec-15:null)
> Agent status update: [id = 5; name = Rack3Host8.lab.vmops.com; old status =
> Disconnected; event = Remove; new status = Removed; old update count = 12;
> new update count = 13]
> 2013-06-27 11:48:36,847 DEBUG [cloud.resource.ResourceManagerImpl]
> (catalina-exec-15:null) Cannot transmit host 5to Enabled state
> com.cloud.utils.fsm.NoTransitionException: No next resource state found for
> current state =Enabled event =DeleteHost
> at
> com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1367)
> at
> com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:999)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:1055)
> at
> org.apache.cloudstack.api.command.admin.host.DeleteHostCmd.execute(DeleteHostCmd.java:83)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:528)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:371)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at
> org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at
> org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at
> org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 11:48:36,855 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
> ===END=== 10.252.120.177 -- GET
> command=deleteHost&id=405585d6-3579-4c8b-82a0-425158ac061a&forced=false&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372358962625
> ===============
> From the Database:
> ===============
> mysql> select id,name,status,uuid,type,private_ip_address,created,removed
> from host where type="Routing";
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | id | name | status | uuid | type | private_ip_address | created | removed |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> | 1 | Rack3Host20.lab.vmops.com | Up | 9b22f04f-f59d-4145-bddb-138027e4cf56 |
> Routing | 10.223.59.66 | 2013-06-27 01:08:20 | NULL |
> | 5 | Rack3Host8.lab.vmops.com | Removed |
> 405585d6-3579-4c8b-82a0-425158ac061a | Routing | 10.223.57.194 | 2013-06-27
> 01:27:34 | 2013-06-27 18:48:36 |
> +----+---------------------------+---------+--------------------------------------+---------+--------------------+---------------------+---------------------+
> 2 rows in set (0.00 sec)
> mysql> select id,name,instance_name,state from vm_instance where host_id=5;
> +----+------------+---------------+---------+
> | id | name | instance_name | state |
> +----+------------+---------------+---------+
> | 4 | san | i-3-4-NTIERR | Running |
> | 5 | r-5-NTIERR | r-5-NTIERR | Running |
> | 6 | san-5 | i-3-6-NTIERR | Running |
> | 7 | san-4 | i-3-7-NTIERR | Running |
> | 8 | san-3 | i-3-8-NTIERR | Running |
> | 9 | san-1 | i-3-9-NTIERR | Running |
> | 10 | san-2 | i-3-10-NTIERR | Running |
> | 11 | san-13 | i-3-11-NTIERR | Running |
> | 14 | san-12 | i-3-14-NTIERR | Running |
> | 15 | san-19 | i-3-15-NTIERR | Running |
> | 17 | san-7 | i-3-17-NTIERR | Running |
> | 18 | san-16 | i-3-18-NTIERR | Running |
> | 21 | san-11 | i-3-21-NTIERR | Running |
> +----+------------+---------------+---------+
> 13 rows in set (0.00 sec)
> Any Agent Operation on the above VMs fail
> 2013-06-27 12:12:12,071 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null)
> ===START=== 10.252.120.177 -- GET
> command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,157 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-24:null) submit async job-35, details: AsyncJobVO {id:35,
> userId: 2, accountId: 2, sessionKey: null, instanceType: VirtualMachine,
> instanceId: 4, cmd: org.apache.cloudstack.api.command.user.vm.DestroyVMCmd,
> cmdOriginator: null, cmdInfo:
> {"id":"c5e5a1c2-f830-4f0b-8250-e548b48681a1","response":"json","sessionkey":"LAHSBLWAfVc8ti8IX/CBeUyvaJQ\u003d","ctxUserId":"2","httpmethod":"GET","_":"1372360378188","ctxAccountId":"2","ctxStartEventId":"170"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 7471666038533,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-06-27 12:12:12,159 DEBUG [cloud.api.ApiServlet] (catalina-exec-24:null)
> ===END=== 10.252.120.177 -- GET
> command=destroyVirtualMachine&id=c5e5a1c2-f830-4f0b-8250-e548b48681a1&response=json&sessionkey=LAHSBLWAfVc8ti8IX%2FCBeUyvaJQ%3D&_=1372360378188
> 2013-06-27 12:12:12,182 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-1:job-35) Executing
> org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for job-35
> 2013-06-27 12:12:12,253 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-35) Destroying vm VM[User|san]
> 2013-06-27 12:12:12,285 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-1:job-35) VM state transitted from :Running to Stopping with
> event: StopRequestedvm's original host id: 5 new host id: 5 host id before
> state transition: 5
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-35) Unable to stop vm, agent unavailable:
> com.cloud.exception.AgentUnavailableException: Resource [Host:5] is
> unreachable: Host 5: Can't find the host
> 2013-06-27 12:12:12,288 WARN [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-35) Unable to stop vm VM[User|san]
> 2013-06-27 12:12:12,294 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-1:job-35) VM state transitted from :Stopping to Running with
> event: OperationFailedvm's original host id: 5 new host id: 5 host id before
> state transition: 5
> 2013-06-27 12:12:12,294 DEBUG [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-1:job-35) Unable to stop VM[User|san]
> 2013-06-27 12:12:12,300 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-1:job-35) Unexpected exception while executing
> org.apache.cloudstack.api.command.user.vm.DestroyVMCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to destroy vm with
> specified vmId
> at
> com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:3386)
> at
> com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:1881)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execute(DestroyVMCmd.java:100)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-27 12:12:12,300 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-1:job-35) Complete async job-35, jobStatus: 2, resultCode: 530,
> result: Error Code: 530 Error text: Failed to destroy vm with specified vmId
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
