zerocarbthirty opened a new issue, #14423: URL: https://github.com/apache/cloudstack/issues/14423
### problem I've built 15 different lab setups of Cloudstack 4.23.0.0 on AlmaLinux 10.2 because I'm trying to document the process of building it in a way that sets up clean without any errors the first time. A lot of the documentation in this url https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html appears to be from the "EL7 era" (for lack of a better term). I have been able to correct this issue before in a couple of my 'multiverse' attempts to get this to install cleanly but it seems as though the resolution is always something different and that feels 'random' to me and I can't really have a lot of 'random' in my deployment. The one single issue that I have every time I do this is when I try to add a KVM host at the end of the 'Add Zone' setup wizard it always does this: 2026-10-16 09:07:24,243 DEBUG [c.c.u.s.SSHCmdHelper] (qtp1209411469-4659:[ctx-7d50b769, ctx-4ab9983a]) (logid:ca4cf6d9) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties GDDRPh8HTmK5DIqM /etc/cloudstack/agent/ SSH command output: chmod: cannot access '/etc/cloudstack/agent/cloud.key': No such file or directory 2026-10-16 09:07:24,244 WARN [c.c.h.k.d.KvmServerDiscoverer] (qtp1209411469-4659:[ctx-7d50b769, ctx-4ab9983a]) (logid:ca4cf6d9) can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 - Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 2026-10-16 09:07:24,244 DEBUG [c.c.h.k.d.KvmServerDiscoverer] (qtp1209411469-4659:[ctx-7d50b769, ctx-4ab9983a]) (logid:ca4cf6d9) can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 - Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 at org.apache.cloudstack.ca.CAManagerImpl.provisionCertificateViaSsh(CAManagerImpl.java:297) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at jdk.proxy3/jdk.proxy3.$Proxy249.provisionCertificateViaSsh(Unknown Source) at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:172) at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:291) at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:881) at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:726) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at jdk.proxy3/jdk.proxy3.$Proxy279.discoverHosts(Unknown Source) at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:155) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:820) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:705) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:414) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:191) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:188) at com.cloud.api.ApiServlet.doPost(ApiServlet.java:147) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:1583) 2026-10-16 09:07:24,245 DEBUG [c.c.r.ResourceManagerImpl] (qtp1209411469-4659:[ctx-7d50b769, ctx-4ab9983a]) (logid:ca4cf6d9) Could not add host at [http://10.1.225.3] with zone [Zone {"id": "2", "name": "Site-ASH", "uuid": "31b0d190-f52c-44f9-9733-40e6e4d3a270"}], pod [HostPod {"id":2,"name":"Site-CMH-1","uuid":"2294eec3-a028-4d57-8fd2-06f113d7789b"}] and cluster [Cluster {id: "2", name: "Site-CMH-1", uuid: "bc234684-68ea-4c40-a81c-3c21b13f1ec5"}] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 - Failed to import certificates into agent keystore via SSH on host: 10.1.225.3]. com.cloud.exception.DiscoveredWithErrorException: can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 - Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:349) at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:881) at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:726) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at jdk.proxy3/jdk.proxy3.$Proxy279.discoverHosts(Unknown Source) at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:155) at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173) at com.cloud.api.ApiServer.queueCommand(ApiServer.java:820) at com.cloud.api.ApiServer.handleRequest(ApiServer.java:705) at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:414) at com.cloud.api.ApiServlet$1.run(ApiServlet.java:191) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:188) at com.cloud.api.ApiServlet.doPost(ApiServlet.java:147) at javax.servlet.http.HttpServlet.service(HttpServlet.java:665) at javax.servlet.http.HttpServlet.service(HttpServlet.java:750) at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to import certificates into agent keystore via SSH on host: 10.1.225.3 at org.apache.cloudstack.ca.CAManagerImpl.provisionCertificateViaSsh(CAManagerImpl.java:297) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215) at jdk.proxy3/jdk.proxy3.$Proxy249.provisionCertificateViaSsh(Unknown Source) at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:172) at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:291) ... 59 more 2026-10-16 09:07:24,245 INFO [c.c.u.e.CSExceptionErrorCode] (qtp1209411469-4659:[ctx-7d50b769, ctx-4ab9983a]) (logid:ca4cf6d9) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions ### versions Cloudstack 4.23.0.0 AlmaLinux 10.2 ### The steps to reproduce the bug 1. Follow the instructions here https://docs.cloudstack.apache.org/en/latest/installguide/management-server/index.html (adapt the instructions for EL10) 2. Follow the instructions here https://docs.cloudstack.apache.org/en/latest/installguide/hypervisor/kvm.html (adapt the instructions for EL10) 3. Create a zone and add a host in the UI ... ### What to do about it? _No response_ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
