it appears that there is a system vm preventing the removal of the host. since the host is down, it cant delete (transition the state) of the downed system vm. possible way to remedy this is to make sure your host is reachable, has storage plugged in and in a good state. zone can/should be disabled, to prevent spawning of new system vm's... then try to destroy the system vm's. There might be a 'grace period' until the vm's are cleaned up from the hosts. Once thats met you can follow it by removing storage and hosts.
On Mon, Apr 8, 2013 at 12:28 PM, Kirk Jantzer <kirk.jant...@gmail.com>wrote: > I want to do some more testing of things, so I'm trying to remove the > existing hosts/clusters/pods/zones, however I can't seem to do so. I'm down > to the last host but get the following: > > No resource adapter respond to DELETE_HOST event for qtwstratus1 id = 13, > hypervisorType is XenServer, host type is Routing > > The host is showing as disconnected in the UI along with all other > important items: > mysql> select id,name,status from host; > +----+------------------------------------+--------------+ > | id | name | status | > +----+------------------------------------+--------------+ > | 1 | qtwstratus1 | Removed | > | 2 | qtwstratus2 | Removed | > | 4 | qtwstratus4 | Removed | > | 5 | qtwstratus5 | Removed | > | 6 | qtwstratus6 | Removed | > | 7 | qtwstratus7 | Removed | > | 11 | s-18-VM | Disconnected | > | 12 | v-2-VM | Disconnected | > | 13 | qtwstratus1 | Disconnected | > | 14 | v-24-VM | Disconnected | > | 15 | s-25-VM | Disconnected | > | 18 | s-26-VM | Disconnected | > | 19 | v-27-VM | Disconnected | > | 20 | v-79-VM | Down | > | 21 | s-103-VM | Down | > +----+------------------------------------+--------------+ > > management-server.log: > > 2013-04-08 15:16:56,713 DEBUG [cloud.api.ApiServlet] (http-8080-6:null) > ===START=== <client_ip> -- GET > > > command=deleteHost&id=40ec48c4-697e-4498-bd3d-f9dd77566f45&forced=false&response=json&sessionkey=7hKixOBlpz0f9%2BRGLFvdx%2B1snRE%3D&_=1365448764976 > 2013-04-08 15:16:56,736 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > ConsoleProxyManagerImpl$$EnhancerByCGLIB$$bdfb2f62 > 2013-04-08 15:16:56,736 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > PremiumSecondaryStorageManagerImpl$$EnhancerByCGLIB$$b376130a > 2013-04-08 15:16:56,736 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > KvmServerDiscoverer$$EnhancerByCGLIB$$e167cbfd > 2013-04-08 15:16:56,736 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > XcpServerDiscoverer$$EnhancerByCGLIB$$18f3e761 > 2013-04-08 15:16:56,736 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Deleting Host: 13 > Guid:5a5efe1c-654b-411c-b4b7-4170ba887716 > 2013-04-08 15:16:56,749 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (http-8080-6:null) Destroying vm VM[ConsoleProxy|v-79-VM] > 2013-04-08 15:16:56,769 DEBUG [cloud.capacity.CapacityManagerImpl] > (http-8080-6:null) VM state transitted from :Running to Stopping with > event: StopRequestedvm's original host id: 13 new host id: 13 host id > before state transition: 13 > 2013-04-08 15:16:56,774 WARN [cloud.vm.VirtualMachineManagerImpl] > (http-8080-6:null) Unable to stop vm VM[ConsoleProxy|v-79-VM] > 2013-04-08 15:16:56,778 DEBUG [cloud.capacity.CapacityManagerImpl] > (http-8080-6:null) VM state transitted from :Stopping to Running with > event: OperationFailedvm's original host id: 13 new host id: 13 host id > before state transition: 13 > 2013-04-08 15:16:56,778 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (http-8080-6:null) Unable to stop VM[ConsoleProxy|v-79-VM] > 2013-04-08 15:16:56,778 WARN [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) There was an error Destory the vm: > VM[ConsoleProxy|v-79-VM] as a part of hostDelete id=13 > 2013-04-08 15:16:56,778 INFO [utils.exception.CSExceptionErrorCode] > (http-8080-6:null) Could not find exception: > com.cloud.resource.UnableDeleteHostException in error code list for > exceptions > 2013-04-08 15:16:56,778 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) There was an error Destory the vm: > VM[ConsoleProxy|v-79-VM] as a part of hostDelete id=13 > com.cloud.resource.UnableDeleteHostException: There was an error Destory > the vm: VM[ConsoleProxy|v-79-VM] as a part of hostDelete id=13 > at > > com.cloud.resource.ResourceManagerImpl.deleteRoutingHost(ResourceManagerImpl.java:1800) > at > > com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.deleteHost(XcpServerDiscoverer.java:701) > at > > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1387) > at > > com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:766) > at > com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34) > at > > com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:847) > at > com.cloud.api.commands.DeleteHostCmd.execute(DeleteHostCmd.java:82) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:544) > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:423) > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:312) > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:64) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:679) > 2013-04-08 15:16:56,779 INFO [utils.exception.CSExceptionErrorCode] > (http-8080-6:null) Could not find exception: > com.cloud.resource.UnableDeleteHostException in error code list for > exceptions > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Adapter XCP Agent says unable to delete host > com.cloud.resource.UnableDeleteHostException: There was an error Destory > the vm: VM[ConsoleProxy|v-79-VM] as a part of hostDelete id=13,There was an > error Destory the vm: VM[ConsoleProxy|v-79-VM] as a part of hostDelete > id=13 > at > > com.cloud.resource.ResourceManagerImpl.deleteRoutingHost(ResourceManagerImpl.java:1805) > at > > com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.deleteHost(XcpServerDiscoverer.java:701) > at > > com.cloud.resource.ResourceManagerImpl.dispatchToStateAdapters(ResourceManagerImpl.java:1387) > at > > com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:766) > at > com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34) > at > > com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:847) > at > com.cloud.api.commands.DeleteHostCmd.execute(DeleteHostCmd.java:82) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:544) > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:423) > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:312) > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:64) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:679) > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > OvmDiscoverer$$EnhancerByCGLIB$$abfe93d0 > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > NetworkUsageManagerImpl$$EnhancerByCGLIB$$34a46271 > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > ExternalDhcpManagerImpl$$EnhancerByCGLIB$$7ded4baf > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > NiciraNvpElement$$EnhancerByCGLIB$$a2b35039 > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > BareMetalDiscoverer$$EnhancerByCGLIB$$9ca73829 > 2013-04-08 15:16:56,779 DEBUG [cloud.resource.ResourceManagerImpl] > (http-8080-6:null) Dispatching resource state event DELETE_HOST to > PxeServerManagerImpl$$EnhancerByCGLIB$$28d62ea9 > 2013-04-08 15:16:56,779 ERROR [cloud.api.ApiDispatcher] (http-8080-6:null) > Exception while executing DeleteHostCmd: > com.cloud.utils.exception.CloudRuntimeException: No resource adapter > respond to DELETE_HOST event for qtwstratus1 id = 13, hypervisorType is > XenServer, host type is Routing > at > > com.cloud.resource.ResourceManagerImpl.doDeleteHost(ResourceManagerImpl.java:769) > at > com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:34) > at > > com.cloud.resource.ResourceManagerImpl.deleteHost(ResourceManagerImpl.java:847) > at > com.cloud.api.commands.DeleteHostCmd.execute(DeleteHostCmd.java:82) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138) > at com.cloud.api.ApiServer.queueCommand(ApiServer.java:544) > at com.cloud.api.ApiServer.handleRequest(ApiServer.java:423) > at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:312) > at com.cloud.api.ApiServlet.doGet(ApiServlet.java:64) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) > at > > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) > at > > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) > at > > org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) > at > > org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) > at > > org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) > at > > org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) > at > org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) > at > > org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) > at > org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) > at > org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857) > at > > org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) > at > org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) > at java.lang.Thread.run(Thread.java:679) > 2013-04-08 15:16:56,781 DEBUG [cloud.api.ApiServlet] (http-8080-6:null) > ===END=== <client_ip> -- GET > > > command=deleteHost&id=40ec48c4-697e-4498-bd3d-f9dd77566f45&forced=false&response=json&sessionkey=7hKixOBlpz0f9%2BRGLFvdx%2B1snRE%3D&_=1365448764976 >
