Thanks Wei, I ran your command: update vm_instance set last_host_id =NULL where last_host_id=12;
This has succesfully marked the host as 'Removed': select * from host where id=12; | 12 | macfkdn17 | 4f2d0b03-0144-4591-923e-2c29bb93038e | Removed | Routing | 10.253.0.57 | 255.255.252.0 | bc:77:e1:26:20:81 | 10.253.0.57 | 255.255.252.0 | bc:77:e1:26:20:81 | NULL | NULL | NULL | NULL | 10.253.0.57 | 255.255.252.0 | bc:77:e1:26:20:81 | NULL | 1 | 1 | 0 | 0 | 0 | iqn.1993-08.org.debian:01:4efdaa48c143 | NULL | KVM | NULL | 98710974464 | NULL | 4.15.1.0 | NULL | NULL | | NULL | 1 | 0 | 0 | 1588922331 | NULL | NULL | 2021-06-18 18:31:27 | 2021-09-03 15:16:12 | 288 | Disabled | NULL | NULL | Disabled| So I then tried to re-add the host (now that the OS has been reinstalled). The add host process just tries for 5 mins but never completes. Some bits that look relevant from management-server.log: 2021-09-09 10:40:45,748 WARN [c.c.a.d.ParamGenericValidationWorker] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Received unknown parameters for command addHost. Unknown parameters : clustertype 2021-09-09 10:40:45,751 INFO [c.c.r.ResourceManagerImpl] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Trying to add a new host at http://10.253.0.57 in data center 1 2021-09-09 10:56:37,914 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ 2021-09-09 10:56:37,914 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Ssh executed failed java.io.IOException: Sorry, this connection is closed. 2021-09-09 10:56:37,915 WARN [c.c.h.k.d.LibvirtServerDiscoverer] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! 2021-09-09 10:56:37,916 WARN [c.c.r.ResourceManagerImpl] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Unable to find the server resources at http://10.250.0.57 2021-09-09 10:56:37,916 INFO [c.c.u.e.CSExceptionErrorCode] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions 2021-09-09 10:56:37,917 WARN [o.a.c.a.c.a.h.AddHostCmd] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Exception: com.cloud.exception.DiscoveryException: Unable to add the host at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:825) at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:611) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at com.sun.proxy.$Proxy192.discoverHosts(Unknown Source) at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:764) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:588) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:321) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:134) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:131) at com.cloud.api.ApiServlet.doPost(ApiServlet.java:98) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1443) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:791) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:602) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:766) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) at java.base/java.lang.Thread.run(Thread.java:829) 2021-09-09 10:56:37,919 INFO [c.c.a.ApiServer] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) Unable to add the host 2021-09-09 10:56:37,920 DEBUG [c.c.a.ApiServlet] (qtp1074389766-476:ctx-b04bca76 ctx-5d37a408) (logid:1fad0881) ===END=== -- POST command=addHost&response=json Any ideas how to proceed next?? Jim
