Running ACS 4.15.1.0 on Ubuntu 20.04.
I have a cloudstack-agent service that isn't able to start. It looks like
libvirtd won't start because of an expired cert:
user@cmp02:/etc/cloudstack/agent# systemctl start cloudstack-agent.service
A dependency job for cloudstack-agent.service failed. See 'journalctl -xe' for
details.
root@bllcloudcmp02:/etc/cloudstack/agent# journalctl -xe
-- A start job for unit libvirtd.service has begun execution.
--
-- The job identifier is 2331.
Aug 22 18:09:56 bllcloudcmp02 libvirtd[13168]: libvirt version: 6.0.0, package:
0ubuntu8.12 (Christian Ehrhardt <christian.ehrha...@canonical.com> Tue, 20 Jul
2021 14:13:56 +0200)
Aug 22 18:09:56 bllcloudcmp02 libvirtd[13168]: hostname: bllcloudcmp02
Aug 22 18:09:56 bllcloudcmp02 libvirtd[13168]: The server certificate
/etc/pki/libvirt/servercert.pem has expired
Aug 22 18:09:56 bllcloudcmp02 systemd[1]: libvirtd.service: Main process
exited, code=exited, status=6/NOTCONFIGURED
I checked and that file is a symlink to the /etc/cloudstack/agent/cloud.crt file which is
in fact expired. I tried to run "Provision Host Security Keys" (I think that is
what I'm supposed to run) but because the agent is down and can't be started it fails:
2022-08-22 17:52:12,809 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Executing AsyncJobVO {id:1105, userId: 2, accountId: 2, instanceType: Host, instanceId: null, cmd:
org.apache.cloudstack.api.command.admin.ca.ProvisionCertificateCmd, cmdInfo:
{"response":"json","ctxUserId":"2","hostid":"7dae2837-270a-4080-934e-59f633b1ec42","httpmethod":"GET","ctxStartEventId":"8442","ctxDetails":"{\"interface
com.cloud.host.Host\":\"7dae2837-270a-4080-934e-59f633b1ec42\"}","ctxAccountId":"2","cmdEventType":"CA.CERTIFICATE.PROVISION"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
result: null, initMsid: 90520733511963, completeMsid: null, lastUpdated: null, lastPolled: null, created: null, removed: null}
2022-08-22 17:52:12,902 DEBUG [c.c.a.ApiServlet] (qtp330739404-491:ctx-6cf11e9c)
(logid:fcee2f79) ===START=== 172.16.44.18 -- GET
jobId=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
2022-08-22 17:52:12,946 ERROR [o.a.c.c.CAManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105 ctx-3460a193) (logid:574cb963)
Host/agent is not available or operation timed out, failed to setup keystore
and generate CSR for host/agent id=21, due to:
com.cloud.exception.AgentUnavailableException: Resource [Host:21] is
unreachable: Host 21: Host with specified id is not in the right state: Alert
at
com.cloud.agent.manager.ClusteredAgentManagerImpl.getAttache(ClusteredAgentManagerImpl.java:574)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:448)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:361)
at
org.apache.cloudstack.ca.CAManagerImpl.generateKeyStoreAndCsr(CAManagerImpl.java:208)
at
org.apache.cloudstack.ca.CAManagerImpl.provisionCertificate(CAManagerImpl.java:189)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy292.provisionCertificate(Unknown Source)
at
org.apache.cloudstack.api.command.admin.ca.ProvisionCertificateCmd.execute(ProvisionCertificateCmd.java:95)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:156)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-22 17:52:13,018 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Unexpected
exception while executing
org.apache.cloudstack.api.command.admin.ca.ProvisionCertificateCmd
2022-08-22 17:52:13,029 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963)
Complete async job-1105, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":"530","errortext":"Failed
to generate keystore and get CSR from the host/agent id=21"}
2022-08-22 17:52:13,037 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Publish async
job-1105 complete on message bus
2022-08-22 17:52:13,038 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Wake up jobs
related to job-1105
2022-08-22 17:52:13,038 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Update db status
for job-1105
2022-08-22 17:52:13,052 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Wake up jobs
joined with job-1105 and disjoin all subjobs created from job- 1105
2022-08-22 17:52:13,067 DEBUG [c.c.a.ApiServlet] (qtp330739404-491:ctx-6cf11e9c
ctx-f920f958) (logid:fcee2f79) ===END=== 172.16.44.18 -- GET
jobId=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
2022-08-22 17:52:13,103 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Done executing
org.apache.cloudstack.api.command.admin.ca.ProvisionCertificateCmd for job-1105
2022-08-22 17:52:13,104 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-45:ctx-48e9c6f9 job-1105) (logid:574cb963) Remove job-1105
from job monitoring
2022-08-22 17:52:15,094 DEBUG [c.c.a.ApiServlet] (qtp330739404-498:ctx-5ae099be)
(logid:6c0898ed) ===START=== 172.16.44.18 -- GET
jobid=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
2022-08-22 17:52:15,261 DEBUG [c.c.a.ApiServlet] (qtp330739404-498:ctx-5ae099be
ctx-9604702f) (logid:6c0898ed) ===END=== 172.16.44.18 -- GET
jobid=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
2022-08-22 17:52:16,126 DEBUG [c.c.a.ApiServlet] (qtp330739404-491:ctx-9e0094c8)
(logid:5dc049fa) ===START=== 172.16.44.18 -- GET
jobId=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
2022-08-22 17:52:16,282 DEBUG [c.c.a.ApiServlet] (qtp330739404-491:ctx-9e0094c8
ctx-3baf6efa) (logid:5dc049fa) ===END=== 172.16.44.18 -- GET
jobId=574cb963-9411-47da-aabd-e0ae2123013c&command=queryAsyncJobResult&response=json
How do I get CloudStack to issue a new certificate?
--
Thanks,
Joshua Schaeffer
