DaanHoogland commented on PR #6892:
URL: https://github.com/apache/cloudstack/pull/6892#issuecomment-1337082470
management:
```
022-12-02 14:36:53,653 DEBUG [c.c.a.ApiServlet]
(qtp262366552-16:ctx-738fd6f7) (logid:944d439c) ===START=== 10.0.35.111 -- GET
apiKey=LIN6rqXuaJwMPfGYFh13qDwYz5VNNz1J2J6qIOWcd3oLQOq0WtD4CwRundBL6rzXToa3lQOC_vKjI3nkHtiD8Q&clusterid=a94
89c75-dbcc-4a51-8081-a76fdff1345b&command=addHost&hypervisor=KVM&podid=1ca6f4d3-fe8b-4eaf-be94-915fcd313a08&response=json&url=http%3A%2F%2F10.0.33.209&username=root&zoneid=ddb1c828-a5eb-4010-83c6-cd9a681a9972&signature=VVRH4DxK1Kow9bs%2B
jnuzGlWRam0%3D
2022-12-02 14:36:53,657 DEBUG [c.c.a.ApiServer]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) CIDRs from which
account 'Account
[{"accountName":"admin","id":2,"uuid":"e1e1ae59-724d-11ed-9f35-1e002d000402"}]'
is allowed to
perform API calls: 0.0.0.0/0,::/0
2022-12-02 14:36:53,659 INFO [o.a.c.a.DynamicRoleBasedAPIAccessChecker]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) Account [Account
[{"accountName":"admin","id":2,"uuid":"e1e1ae59-724d-11ed-9f35-1e002d000402"}]]
is Roo
t Admin or Domain Admin, all APIs are allowed.
2022-12-02 14:36:53,660 WARN [o.a.c.a.ProjectRoleBasedApiAccessChecker]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) Project is null,
ProjectRoleBasedApiAccessChecker only applies to projects, returning API
[addHost] for
user [User
{"username":"admin","uuid":"6ca2d90b-98d3-4e6a-ba3b-328507d9044c"}.] as allowed.
2022-12-02 14:36:53,660 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) RoleService is
enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2022-12-02 14:36:53,661 DEBUG [o.a.c.r.ApiRateLimitServiceImpl]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36) (logid:944d439c) API rate limiting
is disabled. We will not use ApiRateLimitService.
2022-12-02 14:36:53,667 INFO [c.c.r.ResourceManagerImpl]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Trying to add a new host at http://10.0.33.209 in data center 1
2022-12-02 14:36:53,901 WARN [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Failed to authenticate with ssh key
2022-12-02 14:36:53,902 ERROR [c.c.h.k.d.LibvirtServerDiscoverer]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Failed to authenticate with ssh key
2022-12-02 14:36:53,927 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Executing cmd: ls /dev/kvm
2022-12-02 14:36:55,159 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH
command: ls /dev/kvm
SSH command output:/dev/kvm
2022-12-02 14:36:55,171 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
/etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-12-02 14:36:56,054 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-4:ctx-cb611b95) (logid:78fc051a) HA health check
task is running...
2022-12-02 14:36:57,537 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH
command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
/etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
2022-12-02 14:36:57,597 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Executing cmd: sudo
/usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:36:58,864 DEBUG [c.c.c.ConsoleProxyManagerImpl]
(consoleproxy-1:ctx-ca157098) (logid:c345ffc8) Skip capacity scan as there is
no Primary Storage in 'Up' state
2022-12-02 14:36:59,921 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH
command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or
directory
2022-12-02 14:36:59,921 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Executing cmd: sudo
/usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:37:00,055 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-1:ctx-b9a7c1a3) (logid:b4f987fc) HA health check
task is running...
2022-12-02 14:37:02,202 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH
command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or
directory
2022-12-02 14:37:02,202 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
Executing cmd: sudo
/usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
2022-12-02 14:37:04,058 DEBUG [o.a.c.h.HAManagerImpl]
(BackgroundTaskPollManager-2:ctx-4b9064e6) (logid:88af62e1) HA health check
task is running...
2022-12-02 14:37:04,441 DEBUG [c.c.u.s.SSHCmdHelper]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) SSH
command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import
/etc/cloudstack/agent/agent.properties 9MVW4zn3ytOMWKwp /etc/cloudstack/agent/
SSH command output:
chmod: cannot access ‘/etc/cloudstack/agent/cloud.key’: No such file or
directory
2022-12-02 14:37:04,443 WARN [c.c.h.k.d.LibvirtServerDiscoverer]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException:
Failed to setup certificate in the KVM agent's keystore file, please see logs
and configure manually! - Failed to setup certificate in the KVM agent's
keystore file, please see logs and configure manually!
2022-12-02 14:37:04,443 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c)
can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException:
Failed to setup certificate in the KVM agent's keystore file, please see logs
and configure manually! - Failed to setup certificate in the KVM agent's
keystore file, please see logs and configure manually!
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate
in the KVM agent's keystore file, please see logs and configure manually!
at
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
at
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
at
com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
at
com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
at
org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:147)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:144)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-12-02 14:37:04,444 DEBUG [c.c.r.ResourceManagerImpl]
(qtp262366552-16:ctx-738fd6f7 ctx-406aaf36 ctx-5aca3161) (logid:944d439c) Could
not add host at [http://10.0.33.209] with zone [1], pod [1] and cluster [1] due
to: [ can't setup agent, due to
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in
the KVM agent's keystore file, please see logs and configure manually! - Failed
to setup certificate in the KVM agent's keystore file, please see logs and
configure manually!].
com.cloud.exception.DiscoveredWithErrorException: can't setup agent, due to
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in
the KVM agent's keystore file, please see logs and configure manually! - Failed
to setup certificate in the KVM agent's keystore file, please see logs and
configure manually!
at
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
at
com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
at
com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
at
org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
at
com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:330)
at com.cloud.api.ApiServlet$1.run(ApiServlet.java:147)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:144)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:98)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
at
org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
at
org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:516)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
at
org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
at
org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup
certificate in the KVM agent's keystore file, please see logs and configure
manually!
at
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
at
com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
at
com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
... 60 more
etc ...
```
I must admit that I do not directly relate this error to your code either,
but other PRs are passing so I can not attribute it to the environment either.
the kvm actually fails earlier than with the agent:
```
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05
10:08:48.807+0000: 19923: info : libvirt version: 4.5.0, package: 36.el7_9.5
(CentOS BuildSystem <http://bugs.centos.org>, 2021-04-28-13:32:22,
x86-01.bsys.centos.
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05
10:08:48.807+0000: 19923: info : hostname: pr6892-t5375-kvm-centos7-kvm1
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 libvirtd[19923]: 2022-12-05
10:08:48.807+0000: 19923: error : virNetTLSContextCheckCertFile:112 : Cannot
read private key '/etc/pki/libvirt/private/serverkey.pem': No such file or
directory
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 systemd[1]: libvirtd.service:
main process exited, code=exited, status=6/NOTCONFIGURED
Dec 05 10:08:48 pr6892-t5375-kvm-centos7-kvm1 systemd[1]: Failed to start
Virtualization daemon.
-- Subject: Unit libvirtd.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit libvirtd.service has failed.
--
-- The result is failed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
